Language models align with brain regions that represent concepts across modalities
Authors: Maria Ryskina
Greta Tuckute
Alexander Fung
Ashley Malkin
Evelina Fedorenko
Venue: COLM 2025 (2025)
Keywords: LM–brain alignment
fMRI
conceptual meaning
cognitive neuroscience
TL;DR: We find that LMs predict brain signal better in areas that respond more consistently to the same concept encoded in different modalities.

Abstract: Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.

Published as a conference paper at COLM 2025
Language models align with brain regions that represent con- cepts across modalities
Maria Ryskinalt, Greta Tuckute", Alexander Fung", Ashley Malkin', Evelina Fedorenko² Vector Institute for AI MIT
Work done at MIT pariaarrskinn@veeorrrnsittii
Abstract
Cognitive science and neuroscience have long faced the challenge of disen- tangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM-brain alignment and two neu- ral metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) novel measure of meaning consis- tency across input modalities, which quantifies how consistently brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the sig- nal better in more areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning."
consistent areas of the brain, even when these
1 Introduction
Much recent work at the intersection of AI and neuroscience has focused on discovering the similarities and differences between the human brain and increasingly complex and powerful artificial neural models (Oota et al., 2024b; Sucholutsky et al., 2024; Tuckute et al., 2024a). Often, studies compare how these two systems encode information internally-for example, how sentence representations in a language model (LM) align with the responses to the same sentences in a certain region of the brain. Previous work has found correlations between how sentences or narratives are represented in LMs and in the brain's 's language network (Toneva & Wehbe, 2019; Schrimpf et al., 2021; Goldstein et al., 2022; Tuckute et al., 2024b), as well as between image representations in convolutional neural networks and the visual cortex (Yamins et al., 2014; Horikawa & Kamitani, 2017; Conwell et al., 2024). However, as models become more seamless in integrating different modalities, question arises: do these models represent deeper, modality-indee independent conceptual information in a brain-like way?
Recent evidence suggests that such conceptual representations exist in multimodal models (Wu et al., 2025) and that models learn similar representations from different modalities (Merullo et al., 2023; Maniparambil et al., 2024; Huh et al., 2024). However, comparing these representations with the brain is challenging given that the ways in which the brain represents and processes conceptual knowledge remain debated (Kiefer & Pulvermüller, 2012) and there are no clearly delineated conncettrepresenttnng regions". In this paper, we propose new way of localizing concept-representing areas in the brain by using fMRI data collected in a multimodal experiment targeting conceptual processing (Pereira et al., 2018). In this study, participants read text or looked at images representing a particular concept, and their brain responses to these stimuli were recorded. Each concept was presented in three paradigms spanning two modalities (language and vision): in a sentence, (2) as highlighted word in the middle of a relevant word cloud, or (3) as
(1) as highlighted word
1 Our code be found at https: //gs https://ggthuub..m////siiiiiiii higphh...oorykkkiiicceeettttaaaa

---

Published as a conference paper at COLM 2025
picture labeled with the concept word (Fig. 1a). We introduce semantic consistency metric for how consistently a particular brain unit (voxel) responds to the same concept in all three paradigms (64.1), and identify three brain areas that show high semantic consistency (54.2).
Next, we ask the representations from 15 uni- and multimodal transformer LMs of differ- ent sizes are aligned with brain responses in these areas during linguistic and conceptual processing. Our main question is whether LM-based encoding performance correlates with the level of semantic consistency for a given brain region; in addition, we look at the relationship between the encoding quality and the region's selectivity for language. Method- ologically, we use two approaches: (1) using LM features to predict activations in these regions (Fig. 1b), and (2) performing a representational similarity analysis (RSA) to probe the geometric structure of concept representations in the brain and in LMs (Fig. 1c). To preface our key results, all models show significant brain alignment in both the prediction and the RSA analyses. Moreover, high semantic consistency correlates with high predictivity in regions with weak language responses, which suggest that these areas indeed represent non-linguistic conceptual information. Overall, our contributions are the following:
Using an fMRI dataset of brain responses to multimodal stimuli, we define novel metric for measuring semantic consistency in the brain (64.1) and use to find brain regions that represent concepts most consistently, irrespective of paradigm ($4.2);
We evaluate 15 uni- and multimodal transformer models on their ability to predict brain activations in three newly identified semantically consistent regions ($5.3)
and compare the models' representational geometry to the brain's 's ($5.4);
We show that models' predictive performance correlates with our metric of semantic consistency in the brain, both across the whole brain and in the high-consistency regions specifically, including brain regions with a low response to language (56.1);
We find significant representational similarity between the models and the semanti- cally consistent brain regions and show that it further increases when both text and image stimuli are used (S6.2).
Related work
LM-brain alignment A growing body of work compares representations in deep neural network language models to brain imaging data (Karamolegkou et al., 2023; Oota et al., 2024b; Sucholutsky et al., 2024; Tuckute et al., 2024a). Many studies adopt brain encoding approach, predicting brain activations from the model's hidden states (Toneva & Wehbe, 2019; Schrimpf et al., 2021; Merlin & Toneva, 2024) or attention head outputs (Kumar al., 2024). Encoding studies find that best-performing LMs (Schrimpf et al., 2021; Caucheteux & King, 2022) and LMs fine-tuned for certain NLP tasks (Oota et al., 2022a; Aw & Toneva, 2023) tend to be more brain-aligned, and that predictivity increases with scale (Antonello et al., 2023) and with the addition of instruction tuning (Aw et al., 2024). A complementary line of work uses representatio tional similarity alignment (RSA; Kriegeskorte et al., 2008) or direct projection to compare the geometry of the model's sss and the brain's representational spaces (Kaniuth & Hebart, 2022; Yu al., 2024; Li et al., 2024a; Du et al., 2025). Such studies can benefit both NLP and neuroscience: there is evidence that increasing brain alignment can improve model performance (Toneva & Wehbe, 2019) and that models can help scientists elicit targeted levels of neural activity (Bashivan et al., 2019; Tuckute et al., 2024b).
Recent work has explored if lasion- LMs (VLMs) are more brain-aligned than language-only ones (Oota et al., 2022c; Du et al., 2025; Bavaresco & Fernández, 2025), with two studies in particular using the multimodal, concept-focused Experiment data from Pereira et al. (2018) as a testbed (used also in this work). Oota et al. (2022b) perform brain decoding, predicting LM representations of concept words from the brain responses to stimuli in different modalities. Especially relevant to ours is the work of Bavaresco et al. (2024): in an RSA analysis, they find that VLMs capture multimodal knowledge, leading to higher alignment in both language and visual networks. Unlike these studies, do not use known brain networks as the alignment target-wi we identify a novel set of concept- representing brain regions by leveraging the cross-modal nature of the dataset's stimuli.
2

---

Published as a conference paper at COLM 2025
a) Brain data collection
Sentences
Pictures
Word clouds
80 concepts x 4 6
180 concepts x 4
150 concepts x 4
bear
danggrruu
n=17
fMRI
b) Brain encoding
c) RSA
He nit the ball out of the fees.
rollee proactre help has inaree
Pleass clartyy
stimulus
Festie vireed book
bir cage.
fallg tud Broken
The was hepeirerly breken.
brain activation
LM hiddlnn states
brain activation
Fele
help svd
corr
Bid
Figure 1: Brain data collection process for the fMRI dataset (Pereira et al., 2018, Experiment
and the schematics of our two LM-brain alignment evaluations. (a) 17 participants underwent three fMRI scan sessions, one per paradigm (sentences, pictures, or word clouds) to record brain activity when thinking different concepts. Each paradigm presents the 180 concepts in different format: sentences containing the concept word, pictures presented alongside the concept word, and word clouds with the concept word surrounded by related terms. Example stimuli are shown; each concept is represented by 4-6 unique stimuli per session. (b) Brain encoding (55.3): we use the LM representation of the stimulus to predict brain activations in participant viewing the same stimulus. (c) Representational similarity alignment (RSA) ($5.4): we combine all stimuli per concept to obtain a single concept representation from the brain and from the LM. We use them to evaluate pairwise concept dissimilarities in the LM and the brain and correlate them between the two.
Concepts the brain How the human brain represents conceptual meaning is an open question (Kiefer & Pulvermüller, 2012; Frisby et al., 2023), but several streams of scientific evidence suggest that language and semantic/ semantic/conceptut conceptual processing are dissociated in the mind and brain (for details and references, see Reilly et al., 2025, Dissent #1 for event semantics). Therefore, we propose extracting meaning representations not from the language- selective brain regions commonly used in prior brain-LM work, but from the regions that represent meaning independently of whether it is conveyed through text or image. While there is no established method for localizing such regions, brain imaging studies have used visual and linguistic stimuli in parallel to search for amodal semantic processing (Wurm & Caramazza, 2019; Popham et al., 2021; Ivanova, 2022, Ch. 5). Similarly, we use the multimodal, concept-matchi stimuli of Pereira et al. (2018, Experiment 1) to identify regions of interest: we propose novel metric of how consistently a brain area responds to particular cereepts-rggrdleess of whether the concept is shown pictorially, in the context of related single words, or in a sentence context-and select areas where is reliably high (64).
Concepts in LMs While the language models' ability represent concepts without grounding is subject to debate (Bender & Koller, 2020; Piantadosi & Hill, 2022), recent work has found that LMs can learn about concepts like color from text input only (Abdou et al., 2021). Further studies show evidence for the existence of "universal representations" a shared brain-aligned latent space that deep neural models converge on (Hosseini et al., 2024; Chen & Bonner, 2024). Convergence emerges even between models trained on dif- ferent modalities (Maniparambil et al., 2024; Li et al., 2024b), and Huh et al. (2024) argue that models are aligning towards shared representation of reality. Wu et al. (2025) connect these findings to a theory of human cognition (Patterson & Ralph, 2016), showing that LMs develop "semantic hubs" which encode shared meaning across languages and modalities.
3

---

Published as a conference paper at COLM 2025
3 Data
We use the brain data from Experiment of Pereira et al. (2018). They collected fMRI brain recordings 17 participants who perceived the experimental stimuli (text or images). Each stimulus corresponded to a target concept-ong of the 180 single-word labels obtained by performing clustering on a static word embedding space (Pennington et al., 2014). The concept words vary in part of speech (Seafood, Disturb, Willingly, Great) and range from concrete and material (Table) to abstract (Emotion); the list of concepts is included in the Appendix (Table 1). Each stimulus represents concept in one of the three experimental paradigms: as sentence containing the concept word (sentence paradigm, or S), a word cloud with the concept word surrounded by relevant terms (word cloud paradigm, or WC), or an image presented alongside the concept word (picture paradigm, or P). The full dataset of contains six sentences, six images, and six spatial arrangements of the word cloud for each concept (see Figure 6 in the Appendix). 2 The concept word was always highlighted in bold, and the participants were asked to read the text and think about the target word's meaning in relation to the accompanying image or context.
Each participant underwent three separate 2-hour fMRI scanning sessions, one per paradigm, as shown in Figure la. In each session, they viewed 4-6 groups of 180 stimuli (one per concept), in random order. The participant never saw the same exact stimulus more than once: in every new group, the concept was always represented by new sentence, picture, or spatial configuration of the word cloud, depending on the paradigm. Each stimulus was displayed for 3 seconds, followed by a 2-second break.
An fMRI brain recording captures the changes in blood oxygen levels (Blood Oxygenation Level Dependent (BOLD) signal), an indirect measure of neural activity. Spatially, the brain is discretized into 2mm-sized cubical units (voxels). To estimate the activation strength & in each voxel corresponding to each stimulus, we implement a processing pipeline using the GLMsingle toolkit (Prince et al., 2022), with additional upsampling of the BOLD signal time series to align stimuli presentations with the temporal resolution of the scan (2s). Further details about the collection and processing of the fMRI data are provided in Appendix A.
4 Defining and mapping semantic consistency
We use the estimated activation values per stimulus to identify which voxels in the brain consistently respond to the same concepts, whether presented as sentence, a picture, or a word cloud. We propose measure of this conceptual consistency (54.1) and use it to identify brain regions where significantly consistent voxels are likely to be found (54.2).
4.1 Semantic consistency metric
We consider a voxel semantically consistent if it consistently responds strongly (or weakly) to stimuli representing the same concept, regardless of the paradigm (e.g., if it responds strongly to sentences, pictures, and word clouds for the concept Bird but weakly to those for Art). Suppose that the stimuli associated with the concept cj (1 S sis 180) under the paradigm Ω E {s, WC} elicit an average response P/2 E R in a given voxel. We then obtain vectors { 180
Pn E R880 and define the voxel's semantic consistency as follows:
| (Bs.Bo) +r (psr + (BwcrB.))
C 3
(1)
where denotes the Pearson correlation coefficient (Fig. 2a). To apply this measure to set of voxels, we average the response values in over voxels before computing the correlations.
Notably, the words in each concept's word cloud remain the same in each of the six WC stimuli. We use the words 'activation' and 'response' interchangeably to denote the BOLD percent signal change in response to stimulus.
4

---

Published as a conference paper at COLM 2025
a)
Semantic consistency
b)
30%
8%
Bp
concept (Abilty)
0%
c)
ROI 1
2
concept 180 (Willingl)
Figure 2: Semantic consistency and its spatial distribution. (a) The schematic of the com- putation of the semantic consistency measure C. Given a brain activation vector for each of the experimental paradigms (sentences, pictures, and word clouds) over the 180 concepts, we compute Pearson correlation coefficients between each pair of activation vectors and average them. (b) A probabilistic semantic consistency map of the left hemisphere. Each point shows the % of participants whose brain displays significant semantic consistency in that voxel, demonstrating where, average, the semantically consistent brain areas are located. (c) Regions of interest (ROIs) that emerge after overlaying the probabilistic map in (b) with an anatomical segmentation (Glasser et al., 2016).
Brain regions of interest
Brain representations for model-brain alignment are typically extracted from regions en- gaged by the input modality, e.g., the visual cortex for visual stimuli or the language languagenetwort network for linguistic ones. Since aim to explore the effect of representation consistency across paradigms, we define our own brain regions of interest (ROis) in a modality-agnostic way.
First, in each participant's brain we find all voxels whose semantic consistency C is reliably above chance. To account for noise in fMRT recordings, we select those via two independent permutation tests (shuffling Bn and recomputing C) on two separate halves of the data, and select voxels with p < 0.05 in both permutation tests. probabilistic map of such voxels across all participants is shown in Figure 2b: the voxels that show significant C in larger percentage of participants tend to cluster in certain areas of the left hemisphere. For further details this step, including the whole-brain probabilistic map, see Appendix B.1.
We define the boundaries of these areas by overlaying this probabilistic map with popular anatomical segmentation of the brain cortex (the HCP-MMP1.0 atlas; Glasser et al., 2016), which divides each hemisphere into 180 functionally and anatomically distinct areas. After we threshold contiguous clusters of areas by size and by likelihood of high-consistency voxels (full procedure described in Appendix B.2), the three regions of interest (ROIs) are left, marked as ROI 1, and in Figure 2c. ROI located the inferior frontal lobe, and especially ROI 1, which covers parts of the temporal lobe, include areas that are considered to be languate in prior work on brain-LM alignment (Oota et al., 2023; 2024a). ROI 3 contains ventral areas involved in visual processing (Rolls, 2023), which have been used for benchmarking representational alignment in computer vision models (Kaniuth Hebart, 2022). The full anatomical breakdown of each ROI can be found in Table 2 ($B.2).
Brain-LM alignment
We now measure how well brain responses to stimuli in the identified ROIs (Fig. 2c) align with the LM representations of the same stimuli. This section lists the models used in this study ($5.1), outlines how LM representations extracted (55.2), and describes two

---

Published as a conference paper at COLM 2025
methods: brain encoding (predicting brain signal from LM representations; $5.3, Fig. 1b) and RSA (comparing the structure the representational spaces; $5.4, Fig. 1c).
5.1 Models
5.1.1
Language-only models
We experiment with a range of open-weights transformer (Vaswani et al., 2017) LMs from three different series: GPT-2 (Radford et al., 2018; 2019), Qwen-2.5 (Bai et al., 2023a; Yang et al., 2024a;b), and Llama-based Vicuna-1.5 (Chiang et al., 2023; Zheng et al., 2024).
GPT-2 is a series of autoregressive transformer models trained on English text. GPT-2 models are commonly used in brain-model alignment studies and have demonstrated high brain encoding performance (Schrimpf et al., 2021; Tuckute et al., 2024b). We evaluate the small, medium, large, and XL models in this architecture.
Qwen2.5 is family of large multilingual models pre-trained on large dataset with focus on knowledge, coding, and mathematics (Yang et al., 2024b). Both the base pre-trained models and their instruction-tuned version are released; evaluate the 1.5B-, 3B, and 7B-parameter models in both versions.
Vicuna-1.5 1.5 is version of the Llama-2 model (Touvron al., 2023) fine-tuned user-model conversations from ShareGPT. We use the version of Vicuna-1.5 with 7B parameters.
5.1.2 Vision-language models
To incorporate the visual data used in the picture paradigm, we ala experiment with FLAVA (Singh et al., 2022), LLaVA-1.5 (Liu et al., 2024; 2023), and Qwen2.5-VL. (Bai et al., 2023b; Wang al., 2024; Bai et al., 2025) models.
FLAVA is a multimodal model trained to align the text and image representations from two separate ViT encoders (Dosovitskiy et al., 2021) via an extra transformer multimodal encoder. While all other models consider are autoregressive, FLAVA's encoders are trained to optimize the masked modeling objective.
LLaVA-115 is general-purpose visual and language understanding model. It is based the Vicuna LM and additionally trained to take the outputs of visual encoder (CLIP; Radford et al., 2021), projected into the shared representation space through an MLP. We use the 7B version this model in our experiments.
Qwen2.5-VL series of large multimodal models (based on Qwen2.5) optimized for visual understanding, including video comprehension, document parsing, and multilingual text recognition in images. We use the Qwen2.5-VL. models with 3B and 7B parameters.
5.2 LM representations
To get one d-dimensional vector per stimulus (inputted into an LM as per SC.2), we extract hidden states from all model layers and compare multiple pooling methods over the tokens in each image/ /eentenccee For each layer, we take either the last-token hidden state or the mean hidden state over tokens; for FLAVA, we additionally consider the first-token ([CLS]) hidden state. FLAVA also uses independent unimodal encoders, so for multimodal inputs (P) we use their averaged hidden states as well as the multimodal fusion encoder output.
5.3 Experiment Brain encoding
In the brain encoding experiment, we fit regression model that predicts a scalar activation value from a d-dimensional vector representation of a stimulus (Fig. 1b). Following Toneva & Wehbe (2019) and Tuckute et al. (2024b), add ridge penalty since the number of predictors (d) can be quite large. The regression weights are determined as:
W = arg min HER
X + x
(2)

---

Published as a conference paper at COLM 2025
where X E Roxd is the matrix of LM representations of the stimuli seen by the participant (720 - n I 1080) and E IR is the vector of the corresponding brain activations. The quality of fit is evaluated as the Pearson correlation coefficient between the vector of predicted brain responses I = X& and the ground truth activations y.. To obtain an unbiased estimate of this correlation, we perform five-fold cross- coos-validation, fitting the regression model on 80% of the ciimulus-activatiin
pairs at time and measuring the correlation on the held-out 20%; the final estimate is averaged over the five folds. We report the performance only for the layer and token pooling ($5.2) that yield the best predictivity r(ÿ, 17) for the average participant's response in a given brain region, across all folds ($D.3). Following Tuckute et al. (2024b), we tune the regularization hyperparameter E (10 30 1028, } independently for each fold using leave-one-out cross-validation on the training portion (with scikit-learn SC.1).
5.4 Experiment 2: Representational similarity alignment
To further explore differences among the models, we conduct an experiment where we measure the Representational Similarity Alignment (RSA; Kriegeskorte et al., 2008) between each model and each of the selected brain regions. RSA, which compares the pairwise input representation distances in the two spaces (Fig. 1c), is frequently used to evaluate brain- model similarity (Oota et al., 2024b). We focus on the concept representations, averaging the vectors for all sentences, pictures, and word clouds to obtain one model (per-layer) vector
F' R2 and one brain actívation vector &' per concept C₁ 1 S i I 180. The elements of B are responses e in each voxel in chosen brain region A: Bi { We compute two 180 x 180 matrices of pairwise Pearson correlation distances between these vectors: 1 1- r x and 1 - rrbb, B/). Finally, measure the Spearman correlation between the lower triangular portions of these matrices to evaluate how similarly this brain region and this model layer represent the 180 concepts. As before, we repeat this for each model layer and token pooling method and report only the results for the best setting per model.
5.5
Neural metrics
We investigate how LM-brain alignment correlates with two neural measures: (1) semantic consistency of a brain area (as described in $4.1) and (2) the selectivity of this area for language processing, defined as the response to well-formed sentences compared to per- ceptually matched control condition. Specifically, we leveraged data from an independent language localizer task (Fedorenko et al., 2010) where brain activitation is compared between two types of stimuli: English sentences and unconnected sequences of nordsords (e.g., REDENTION ZOOD CRE .). The "sentences > non-words" contrast has been shown to reliably identify areas of the brain that are engaged in linguistic processing but not other functions (Fedorenko et al., 2011; Benn et al., 2023; Chen et al., 2023; see Fedorenko et al., 2024 for review). We quantify the language selectivity measure the difference between voxel's activations in the two conditions (ABSentenees,on-wwooll separately in each participant.
6 Results
6.1
More semantically consistent voxels are better predicted by LMs
Brain encoding experiments measure LM predictivity, i.e., the correlation between the voxel activations predicted from the LM representations and the ground truth activations ($5.3). For the sentence and picture paradigms, we predict the brain activations for each stimulus individually (n=720-1080 stimuli per participant), but average the brain activations for word clouds since they contain the same words for the same concept (n=180). This section reports all results averaged over the appropriate models (all models for S and WC paradigms, only vision-language models for P) since we did not see strong differences between individual models (individual plots included in Appendix D.4; see $7 for discussion).
First, we verify that semantic consistency influences predictivity across the whole brain cortex. Figure shows the mean (over LMs and participants) predictivity across the 360
7

---

Published as a conference paper at COLM 2025
Sentence paradigm
Picture paradigm
Word cloud paradigm
Mean LM prodictivity
Mean LM predictivity
Mean LM predictivity
Glasser areas
0.3-
03
0.3
In ROI in ROI 2 in ROI other
0.2
02
02
0.1
0.1
0.1
0.0
0.-
0.0
0.000
0,075 Semantic consistency
0.025
0.060
0.100
0.000
0.025
0,050
0,075
0,100
0,000
0.025
0,050 o 0.675 0,100
Semantic consistency
Semantic consistency
Figure 3: Predictivity vs. semantic consistency in Glasser et al. (2016) anatomical areas (both hemispheres). Each point corresponds to one area, and the areas that fall in the chosen semantically consistent ROIs (64.2) are marked by shape and color. Error bars show standard error over participants. All paradigms show correlation between predictivity and semantic consistency, though for pictures it is skewed by visual cortex areas (circled).
Sentence paradigm Controlled for language
Picture paradigm
Word cloud paradigm Controlled for language
Controlled for consletency
Controlled for language
Controlled consislency
Controlled consletency
Predictivty
ROI
Prediclvity
22
P
6.
Predictiity
ROI
3 Quartile (cons.)
2 Quartlle (lang.)
Quartlia (lang))
Quartile (cons.)
Quartile (lagg))
Quartile (cons.)
Figure 4: Mean LM predictivity by quartile for each ROI and paradigm. Columns 1, and peeediiiiii each ROI changes across voxel quartiles by semantic consistency, 5 show how predictivity
with each red line corresponding to one language selectivity quartile. Columns 2, 4, and show how predictivity changes across voxel quartiles by language selectivity, with each blue line corresponding to one semantic consistency quartile. The thickness of the line corresponds to the quartile (thicker=higher) and the error intervals show standard error across participants. While ROI 1 and ROI 2 (rows and 2) show a positive correlation with both the semantic consistency and the language selectivity (albeit to a lesser extent), the predictivity in the ventral ROI 3 does not correlate with the language selectivity.
anatomical areas (180 in each hemisphere; Glasser et al., 2016) for each of the three paradigms. We see strong positive correlation between the semantic consistency of an area* and how well the activation in it can be predicted by LMs (ris) = 0.79, r r|wc] 0.74). The correlation is lower for the picture paradigm (r[p) 0.17) because of a cluster of visual cortex areas (circled in yellow): they encode images (hence the high VLM alignment) but not necessarily concepts. Taken together, these findings show that a brain region is better predicted responds more consistently to concepts, irrespective of modality and paradigm.
Second, we evaluate how the brain encoding performance in our three ROIs correlates with the two brain metrics of interest (55.5). Each participant's brain voxels in each ROI are divided into bins (quartiles) by either semantic consistency (1 < bc < 4) or language selectivity (1 < by < 4), resulting in 16 (bc, by) bins total. Each plot in Figure 4 keeps one of these metrics fixed while varying the other: for example, in columns 1, 3, and 5 each red
*Measured as probability of significantly consistent voxels ($B.1) to match $4.2. Overall trends also hold when using the the raw value of C (SD.1) or adjusting for inter-participant noise ceiling (SD.2).
8

---

Published as a conference paper at COLM 2025
Language-only models
Vision-language models
(oxt images)
Pesult (rext only)
Cetieation
0.103
Permuted beseline
0.102
ROI
0.060
0.060
0.025
Plesult (text images]
0.125
Result (pect only)
Permuted baseline
0.125
ROI
2
Carreleto
0.075
0.075
0.050
0.002
0.000
0.000
(boxt
(text
Conglation
0.100
Permuted bascline
0.100
ROI
3
0.08
0.77
0.025
0.025
000
0.000
GPT-2
OPT-2 Medium
GPT-2 XL
Qwen2 5-1 1.58
Owen2.5: 2-5--1 1.58-
Owen2.5-300 3B
Qwen2.550 5-38- Instruct
Owen2.5-22 5-BB
Vizuna-1 1amma-1-.55. 1557B
FLAVA
Owen2 S-VL 38 instruct
Owan2 5.V.. 78 re-instruut
LLaVA-1 1.5-B
5wen2.5 Instruct
GPT-2
Figure 5: Concept-level RSA for each LM and brain ROL. RSA quantifies correlations
between pairwise concept distance matrices. Each concept's representations are averaged over stimuli (sentences and word clouds for text-only condition, all paradigms for text + image). Shuffled baseline included for comparison. Error bars show SEM across participants.
line corresponds all voxels of the same by, while the points on the line represent voxels in b1,, (2, bL), (3, bL), and (4, bt) respectively. Similarly, the plots in columns 2, 4, and group lines by bc, and the x-axis steps correspond bt. (1,2,3,4] 4] respectively. The y-axis in each plot shows the predictivity, averaged over participants and LMs.
In all ROIs and paradigms, predictivity rises monotonically across semantic consistency quartiles bc while by is held fixed (red lines in columns 1,3, 5; mean r = 0.40 + 0.01 ). The correlation with the language quartile by. when controlling for bc is less clear: while there is some increase in ROI and 2 for text-based paradigms (blue lines in rows 1, 2, columns 2, 6; mean r 0.26 0.04) ), ROI (row 3, columns 2, 4, 6; mean r 0.01 0.02) and the picture paradigm overall (column 4; mean involved in visual but not language processing, demonstrates that semantic consistency drives predictivity even decoupled from language: rc = 0.33 + 0.03, r = 0.01 + 0.02.
- -0.11 + 0.04) display no such dependency. ROI 3,
6.2 LMs and VLMs share representational geometry with semantic brain regions
Figure 5 shows the RSA correlation between each model and each ROI, reported for the most aligned layer in each model. We additionally include a baseline which we shuffle the brain's concept representations [ so that the pairwise concept distances are not matched between the brain and the model; each baseline is reported for own best layer.
In the three ROIs associated with high semantic consistency (rows in Figure 5), the alignment in all models significantly higher than the baseline. We do not see a clear trend for model size (within the GPT-2 or Qwen2.5 families; cf. Schrimpf et al., 2021) or noticeable effect of instruction tuning (between Qwen2.5 and Siven..5-Instru. models of the same size; cf. Aw et al., 2024). While language-only models (left set of bars) only represent the textual stimuli (s and WC), for vision- vision-language models (right set of bars) we compare the same setting with an all- all-paradigm average. The text-only performance is comparable in VLMs and their base

---

Published as a conference paper at COLM 2025
LM counterparts (LLaVA vs. Vicuna, Qwen2.5-VL vs. Qwen2.5). Interestingly, the addition of multimodal stimuli increases aligmment, most notably in the ventrotemporal ROI 3-a region adjacent to areas associated with high-level vision (e.g,, Kanwisher et al., 1997) as well as the visual word form area and the basal language areas (Li et al., 2024c).
7 Discussion and conclusion
We evaluated model-brain alignment for 15 transformer language and vision-lane rision-language models in brain encoding experiment. To do so, we introduced new metric that identifies brain voxels with consistent responses to conceptual content across different paradigms, based on fMRI data from multimodal stimuli. We show that the more concept-consistent the voxels are, the better they are predicted by LM representations. In line with prior work (Ayesh et al., 2024), we also find that LM predictivity is correlated with language selectivity in the regions overlapping with the canonical language areas of Fedorenko et al. (superior temporal ROT 1) or adjacent to them (inferior frontal ROT 2).
Aiming to extract modality-independent conceptual representations of the stimuli from the participant's brain (rather than purely linguistic/ / /visual ones), we target a novel set of ROIs in our alignment experiments. We focus on three brain regions that show the most consistent preferences for certain concepts, regardless of presentation paradigm (as measured by our proposed semantic consistency metric). These regions are distinct from the established brain networks typically used to evaluate LM-brain alignment, such as the language network (Fedorenko et al. 2010; evidenced by two of the ROIs showing little to no response to the language localizer, see SB.3). The temporal ROT overlaps both with the language network and with the areas where evidence of amodal semantic processing was found previously (Wurm & Caramazza, 2019; Popham et al., 2021; Ivanova, 2022, Ch. 5) hypothesize that ROI may serve as a gateway between the language system and the more abstract semantic areas. For consistency with prior work, we include a comparison of the brain encoding performance in our ROIs and in the language network parcels (SD.5).
We do not see strong differences in brain encoding performance between individual models. We attribute that to the flexibility of our brain encoding pipeline, based on that of Tuckute et al. (2024b): it not only chooses the most predictive layer for each model, ROI, and paradigm, but also tunes the regularization hyperparameter individually for each cross- validation fold at inference time. While it yields the best performance for each model, it obscures the differences between them, so we perform an additional comparison using RSA. We find significant alignment between the representational spaces of all models and the semantically consistent brain regions, but do not observe the trends noted in prior work: in our experiment, RSA alignment does not increase from smaller to larger models in the same architecture (Schrimpf et al., 2021) or with additional instruction tuning (Aw et al., 2024).
Past work has found that responses to images in high-level visual cortical creas-which overlap with the ventral ROI 3-are successfully predicted from LM embeddings of their descriptions (Doerig et al., 2022). Conwell et al. (2023) show that much of this alignment is explained by the concepts (objects and agents) present in the image. Together with our consistency evaluation, these results suggest that certain conceptual information is retained in these regions-ary and stronger LM alignment with semantically consistent brain areas can be viewed as evidence for these models' ability to capture cross-modal conceptual meaning.
Acknowledgments
We would like to acknowledge the Athinoula A. Martinos Imaging Center at the McGovern Institute for Brain Research at MIT and its support team (Steve Shannon and Atsushi Taka- hashi). We thank Francisco Pereira and Juniper Pritchett for developing the experimental materials and collecting, preprocessing, and analyzing the fMRI data. EF was supported by NIH award NS121471 from NINDS and research funds from the McGovern Institute for Brain Research, MIT School of Science, the Simons Center for the Social Brain, and MIT's Quest for Intelligence. We thank Ev members for help with data processing and visualization and the anonymous reviewers for their valuable feedback.
10

---

Published as a conference paper at COLM 2025
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? A case study in color. In Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 109-132. Association for Computational Linguistics, 2021. URL https://aclantholoogy. how.aaannntooooooor//2222222 org/2011 http//222..coo11.11.
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fMRI. In Advances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023, URL http:/tt///rrooceiigg..oot
neurips cc/paper http:ppppeffl//ppppppeee www.ff//lpeeee//////// fil2/553445448225522772223233333 ottp://eeeasaaaacaaaaoaaorrrrrrr pdf.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives im- proves brain alignment. In The Eleventh International Conference on Learning Representations, 2023. URL httttre///peervvieww net http://nee/tte///i/c///.AAAAAA//
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosse- lut. Instruction-tuning aligns LLMs to the human brain. In First Conference on Language Modeling, 2024. URL https: hnttps::/oppe/es /ttp::///pppeeeeeeeeeeeee net/pdf? ioddfidnXXNNOAbb.
Eyas Ayesh, Shailee Jain, Josleen St Luce, Alexander Huth, and Anna A. Ivanova. The language network occupies a privileged position among all brain voxels pre- dicted by a language-based encoding model. In Conference on Computational Cog- nitive Neuroscience, 2024. URL https: /2224.2222242// ccneuro org/ /rg/pdf/ wwwp//////aaera....////eeeeaaeee Paper-authored. -uthored---Lanuaaee wewengeeeetwoo--ooooo wwwpwoooo--cccsso-ooooeeeee-eeee pri wervvileeed-pono poonnddppsiioinn .ddf.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayi- heng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and lianhang Zhu. Qwen technical report. ar preprint ar arii:2239..16009 2023a. URL https://ar /arss///arxvv..o/ https///arxi..or//ab///2000
16689.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: versatile vision-language model for understanding, localization, text reading, and beyond. Xrv preprint Xiie:2308.12966 2023b. URL kttps://arxivv www./br/////eeeeoo/oooooeeesrr/e 2208.. 12966.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL. technical report. arXiv preprint ar 1332022502.13923, 2025. URL https://arxiv. org/abs/2502. 23000/
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 36+(6439)eaav99466 2019. URL https: http:///w/wccecceeeee/ee org/doi/ 10. 2w.1112//ccience..m aav9436.
Anna Bavaresco and Raquel Fernández. Experiential semantic information and brain alignment: Are multimodal models better than language models? In Proceedings of the 29th Conference on Computational Natural Language Learning, pp. 141-155. Association for Computational Linguistics, 2025. URL
org. 202g/222..00 2ttp:/222.coo1111..0.. con11-1.10.
Anna Bavaresco, Marianne de Heer Kloots, Sandro Pezzelle, and Raquel Fernández. Mod- elling multimodal integration in human concept processing with vision-aandlaanguage models. ar Xiv preprint arXit:2407.17914 2024. URL
279bb//////..11114444 17914.
11

---

Published as a conference paper at COLM 2025
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Anmull Meeting of the Association for Computational Linguistics, pp. 5185-5198. Association for Computational Linguistics, 2020. URL hacps//hoolattooog....ooooo org.oogg// acl-main. 463/-
Yael Benn, Anna A. Ivanova, Oliver Clark, Zachary Mineroff, Chloe Seikus, Jack Santos Silva, Rosemary Varley, and Evelina Fedorenko. The language network not engaged in object categorization, Cerebral Cortex, 33(19):00380-10400) 2023. URL https://acaaemicc pdf.
oup. www.c///////////eeeeceee petooo//rticllepppeeee /ttpp////////11111/////1111 0t111///08880/7775777661/11a
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communiicatinn Biology, 5(1):134, 2022. URL https:
//www. www..nature.comm http:///ee../crriieee////////-00
Xuanyi Chen, Josef Affourtit, Rachel Ryskin, Tamar I. Regev, Samuel Norman-Haainnere, Olessia Jouravlev, Saima Malik-Moraleda, Hope Kean, Rosemary Varley, and Evelina Fe- dorenko. The human language system, including its inferior frontal component in "Broca's area," does not support music perception. Cerebral Cortex, 33(12):7904-7929, 2023. URL https://academic c. oup. /ttp:///errrrrrrrrcccc//////////
pdf.
Zirui Chen and Michael F. Bonner. Universal dimensionn of visual representation. arXiv preprint arXio:2408.1220 12804, 2024. URL htt.:ts///arvvv.org//bb///// 12844 12824.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing GI'T-4 with 90%* ChatGPT quality, March 2023. URL https://mmsss.orr/blo////// org/blogg///22--333333 30-3-u/a..
Colin Conwell, Jacob Prince, George Alvarez, and Talia Konkle. The unreasonable ef- fectiveness of word models in predicting high-level visual cortex responses to natu- ral images. In Conference on Computational Cognitive Neuroscience, 2023. URL https:
//2023, www.uro. orthooo.rg/////ee wwwneuro.orggppccceeeeeeee ptocceedings//00055..... http:///c...ooor/orreeeee//e//e/
pdf7s=W&pn=1642.
Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15(1):9383, 2024. URL https://wwwt //auue
htmm/articcees//147---0000 53147-4-
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions objective of human vision. ar ar preprint ar Xix::2209.117371 2022. URL https://arxiv htsp/bbs/20000 117377
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 x 16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. wwtpeeewwwwwwwwwweeeeeeeeeeeeeee
URL
Changde Du, Kaicheng Fu, Bincheng Wen, Yi Sun, Jie Peng, Wei Wei, Ying Gao, Shengpei Wang, Chuncheng Zhang, Jinpeng Li, et al. Human-like object concept representations emerge naturally in multimodal large language models. Nature Machine Intelligence, pp.
1-16, 2025. URL httph:://////// //www//wwwwww//// nature. com/ att.iclee//4225502555555 0205566-025-0000 049-z.
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Castañinn Susan Whirield-Gabrieli, and Nancy Kanwisher. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Neurophyssiology, 104(2::177-11194 2010. URL https://journas whw.iooooyyyyygggoyoooooo ottp:/oollpoee////022112a////o/ jn. 00032.
Evelina Fedorenko, Michael K. Behr, and Nancy Kanwisher. Functional specificity for high-level linguistic processing in the human brain. Proceedings of the National Academy of Sciences, 108(39):1622--16433, 2011. URL https: / //www/ pnas. wrtps/i/a///s/s/////////o////os. p 1112937108.
12

---

Published as a conference paper at COLM 2025
Evelina Fedorenko, Anna A. Ivanova, and Tamar I. Regev. The language network as natural kind within the broader landscape of the human brain. Nature Reviews Neuroscience, 25(5): 289-312, 2024. URL
nature. cwm/ ctt.m//atiicelee/1//----00000
00802-4.
Saskia Frisby, Ajay D. Halai, Christopher R. Cox, Matthew A. Lambon Ralph, and Timo- thy T. Rogers. Decoding semantic representations in mind and brain. Trends in cognitive sciences, 27(3):258-22 281, 2023. URL https: /ttps::/www// cell. com/treeds//com Selve-ssceeness
Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smithh and David C. Van Essen. A multi-modal parcellation of human cere- bral cortex. Nature, 536(7615) 2016. URL satps://wwi nature. com/ar her/aartclless nature18933.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3): 369-380, 2022. URL https::/ww..natt http://wwwwatu...aeee// cwm/cr/////ooooo www./o/tti/eeoo.ecooooooooeettee wwtaaartiilssssseeeeeee 593--2222292 01026-4.
Charles Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Serard-Marchantt Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis F. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, 2020. 2020.
URL https:///oi..ooo /rrg://///o...//.../.. 01338/41588--22 000-886-220-64--22--2
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communicattions, 8(1):15037, 2017. URL
https://www.rr nature. htt.tureeccommarreeeee/n/nnooooo 1537..
Andreas Horn. HCP-MMP1.0 projected on MN12009a GM (volumetric) in NIfTI format, 0 projected on.
August 2016. wrF.://ww.....tttetrr.ciii//T///
Eghbal Hosseini, Colton Casto, Noga Zaslavsky, Colin Conwell, Mark Richardson, and Evelina Fedorenko. Universality of representation in biological and artificial neural networks. bioRxiv, 2024. URL https: www.oiiwwwrg.ooneet////ooooooeee 1101 /2024.222422 12.26.
629294v1.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024. URL html.
https://proceediing wwa.nng.......... wtt.perees///hhhee/eee
Anna Alexandrovna Ivanova. The role of language in broader lumman cognition: evidence from neuroscience. PhD thesis, Massachusetts Institute of Technology, 2022. URL https:
//dspace mit. www..t.dddaaanaee/eeeee 1/122..//17444 1/147484.
Philipp Kaniuth and Martin N. Hebart. Feature-reweighted representational similarity analysis: A method for improving the fit between computational models, brains, and behavior. Neurolmage, 257:119294, 2022. URL https:///oi. org/10. 1016/ neuroimage. 2022. 119294.
Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The fusiform face area: module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11): 4302-4311, 1997. URL https: htttps::/www... jneurossiii.eiiiiiiii.o.... wttprescii...rcooenettneeeem /tt////ntt//////////////// 2/t///11//1/4.2...
13

---

Published as a conference paper at COLM 2025
Antonia Karamolegkou, Mostafa Abdou, and Anders Søgaard. Mapping brains with language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 9748-9762 Association for Computational Linguistics, 2023. URL https:
//aclanthology.. org/ org/2023. wttp///2...ininnne......... findings-acl. 618/.
Markus Kiefer and Friedemann Pulvermüller.
Conceptual representations mind and brain: Theoretical developments, current evidence and future directions. Cortex, 48 (7):805-825, 2012. URL htops://www.wr https::/www.scienceeirr scieccedreecc..rtt.....mm wwtpcccioc//c/c/cccccccoococceee aewaaretieeeaaeeeeeeeee abs/pii/
00010945211001018
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational sim- ilarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:249, 2008. URL https://www. frontiersin org/journals/ trontierrsi..orrgjjornsss/// systems-neerosciencee www.areeereeeee.. wtt.clee//..33///eueeee 3389/neuro. 06 .tw.:.............. folll
Sreejan Kumar, Theodore R. Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A. Norman, Thomas L. Griffiths, Robert D. Hawkins, and Samuel A. Nastase. Shared functional specialization in transformer-based language models and the human brain. Nature Communiications, 15(1):5523, 2024. URL https: nature. http://wwnntuure.ee//rrir..eeeee
$41467-924-49173-
Jiaang Li, Antonia Karamolegkou, Yova Kementchedjhieva, Mostafa Abdou, and Anders Sogaard. Structural similarities between language models and neural response measure- ments. In Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations, volume 228 of Proceedings of Machine Learning Research, pp. 346-365, PMLR, html.
16 Dec 2024a. URL httpps//a /tapps://pron http:///ooccedieeeeeee wetpeeediingnmmm... mlr press/v228/12s
Jiaang Li, Yova Kementchedjhiieaa, Constanza Fierro, and Anders Søgaard. Do vision and language models share concepts? A vector space alignment study. Transactions of the Association for Computational Linguistics, 12:1232-1249, 2024b. URL https://dot org/10 / 1162/tac1 00099..
Jin Li, Kelly Hiersche, and Zeynep M. Saygin. Demystifying visual word form area visual and nonvisual response properties with precision fMRI. iScience, 27(12), 2024c. URL https://doi. org/10. 1016/j isci. 2024. 1124.111440 111481.
Benjamin Lipkin, Greta Tuckute, Josef Affourtit, Hannah Small, Zachary Mineroff, Hope Kean, Olessia Jouravlev, Lara Rakocevic, Brianna Pritchett, Matthew Siegelman, et al. Probabilistic atlas for the language network based precision fmri data from; 800 individuals. Scientific data, 9(1):529, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tun- ing. In Advances in Neural Information Processing Systems, volume 36. Curran As- sociates, Inc., 2023. URL
neurips. pdf.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL hwt.aaae.w...tt.ccaaa......onnna
wtwppppeeaaaccceeeeeeeeeeeeeeeee thecvf. com/ /CV22//
pdf.
Kyle Mahowald and Evelina Fedorenko. Reliable individual-level neural markers of high-level language processing: A necessary precursor for relating neural variabil- ity to behavioral and genetic variability. Neuroimage, 139:74-93, 2016. URL https:
//doi.org/1e. 1016/j neuroimage. 2016.......2..... 02.06..5..7
Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E. O'Connor. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024. URL http:///pppps///aaaaaaaaaa Maniparambil Do Vision and Language Encoders Represent the World Simi larly CVPR 2024 paper. paper.pdf.
https: / http:s//ppeecccsss... thecvf. com/
14

---

Published as a conference paper at COLM 2025
Wes McKinney. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python Science Conference, pp. 56-61, 2010. URL https://proceedingg scipy. org/ articles/Majora-990 92bf
Gabriele Merlin and Mariya Toneva. Language models and brains
align due more than next-word prediction and word-level information. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18431-18454. Association for Com- putational Linguistics, 2024. URL htw.httts//cclanthhhe http///llcllttooogy.y.o.. org/2024. .1114.....llppp.. emnlp-maing 1024/.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp:////pppeeeeeee/eemeeee wwwpiiiw.ne//ppeee httt//ddddiid-tttttoob555
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi R. Surampudi. Neural language languaa taskonomy: Which NLP tasks are the most predictive of fMRI brain activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220-3237, Association for Computational Linguistics, 2022a. URL naacl-main. 235/.
www.oog...rr//// org/20222
Subba Reddy Oota, Jashn Arora, Manish Gupta, and Raju S. Bapi. Multi-view and cross- view brain decoding. In Proceedings of the 29th Infernational Conference on Computational Linguistics, pp. 105-115. International Committee on Computational Linguistics, 2022b. org/ 22t. wow.ng-1.
URL https::
Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S. Bapi. Visio- linguistic brain encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 116-133. International Committee on Computational Linguistics, 2022c
URL htttps://aallanthllllo orr. 2022.022222222 cow.:g-...nggngoe
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic prop- erties in brains and language models. In Advances in Neural Information Processing Systems, volume 36, pp. 18001-18014, 2023. URL https: https://proceedinngs neuriis. cc ec/paper__ _files// (atper//222/ffeeeeeeeee 5
pdf.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models
lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8503-8528. Association for Computational Linguistics, 2024a. URL https: http:s:s///clanthologg.oooo wrw.//24..../// wtt.8224.ccl1110 acn-long. 462/.
Subba Reddy Oota, Zijiao Chen, Manish Gupta, Bapi Raju Surampudi, Gaël Jobard, Frédéric Alexandre, and Xavier Hinaut. Deep neural networks and brain alignment: Brain en- coding and decoding (survey). Transactions Machine Learning Research, 2024b. URL https://openreview.. wtt.:t//ddf/iddxiiiihhhbbb
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chin- tala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/prooceedings www.seeeddinnsnnnniiii..... courips..c/pappt. files//paerr2219/ff11e pdf.
b
Karalyn Patterson and Matthew A. Lambon Ralph. The hub-and-spoke hypothesis of semantic memory. In Neurobiology of Language, Pp. 765-775, Elsevier, 2016. URL https: 24877942080614
www.sscienceirred. www.eeccdrrrett.ccocciicnee/ee/e
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Van- derplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and
15

---

Published as a conference paper at COLM 2025
Edouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, (2(85))2825-2884) 2011. URL httt:://fmll..orr/apppprss//////
1a. html.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Association for Computational Linguistics, 2014. URL https://aclaanthology org/D14-1162/.
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward universal decoder of linguistic meaning from brain activation. Nature Communicatiios, 9(1):963, 2018. URL https://wwet nature. com/articles//41144444/ 460-0//-///1117--1-----------0-0
Steven T. Piantadosi and Felix Hill. Meaning without reference in large language models. In NewrlPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI), 2022. URL https: // wwr.iv.org/abs//2///88 02957.
Sara F. Popham, Alexander G. Huth, Natalia Y. Bilenko, Fatma Deniz, James S. Gao, Anwar O. Nunez-Elizalde, and Jack L.. Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24(11):1628-1666 2021.
URL https: wwwure htt.e//rrrrriicee///11333--0000 022-0991-66...
Jacob Prince, lan Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife, 11:e77599, 2022. URL
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improv- ing language understanding by generative pre-training. Technical report, Ope- nAI, 2018. URL https://cdn. openai. language understanding paper. pdf.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAT, 2019. URL https://cdn, hpwpa/acccooooiiiiii.ooaaa unsupervised mmul wwwuuuitsskllee. tew....ee.eeeerssss......ee .pd.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research. PMLR, 2021. URL v139 htt///rfforr2.aaffm
languaae supervision. In
mlr. mll..p .rres//
Jamie Reilly, Cory Shain, Valentina Borghesani, Philipp Kuhnke, Gabriella Vigliocco, Jonathan E. Peelle, Bradford Z. Mahon, Laurel J. Buxbaum, Asifa Majid, Marc Brys- baert, Anna M. Borghi, Simon De Deyne, Guy Dove, Liuba Papeo, Penny M. Pexman, David Poeppel, Gary Lupyan, Paulo Boggio, Gregory Hickok, Laura Gwilliams, Leonardo Fernandino, Daniel Mirman, Evangelia G. Chrysikou, Chaleece W. Sandberg, Sebastian J. Crutch, Liina Pylkkänen, Filing Yee, Rebecca L. Jackson, Jennifer M. Rodd, Marina Bedny, Louise Connell, Markus Kiefer, David Kemmerer, Greig de Zubicaray, Elizabeth Jefferies, Dermot Lynott, Cynthia S. Q. Siew, Rutvik H. Desai, Ken McRae, Michele Diaz, Mari- anna Bolognesi, Evelina Fedorenko, Swathi Kiran, Maria Montefinese, Jeffrey R. Binder, Melvin J. Yap, Gesa Hartwigsen, Jessica Cantlon, Yanchao Bi, Paul Hoffman, Frank Garcea, and David Vinson. What we mean when we say semantic: Toward a multidis- ciplinary semantic glossary. Psychonomic Bulletin & Review, 32(1):243-280, 2025. URL https://link: springer com com/article/1e. 0tt.:/10...77// 3758/s13423-0244 0245566-777
Edmund T. Rolls. The ventral visual system. In Brain Computations and Comnetivity, Oxford University Press, July 2023. ISBN 9780198887911. URL https:///oo org/10. 10. 1093/050/ 97801 98887911 1.
16

---

Published as a conference paper at COLM 2025
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 11((4))e21106646118 2021. URL https://wwi www..ppa...... org org/doil 1073 pnas. 2105646118.
ful1/10. (011//0.1073//0
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision align- ment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022. URL https: //ppaacccs/pe thecvf. /o//////ttt//////////22// papers/ httpa//euuuuatilaaaaaaaaaaaaaaan
pdf.
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Christopher Cueva, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L.. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nathan Cloos, Nikolaus Kriegeskorte, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O'Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Rober Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. arv preprint Xiv 2310. 1318,, 2024. URL https: /ttp///axxvvvvo/ wwt.:/oo///bb/////ee 13018.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: //apper//ppprrssse nips. 09a8ecc23181116777622232--888883 Absttrattc.c..asttt........a.... html..
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arviv preprint ar arXie:2307.09288 2023. URL https: http:///xxvvv.vv wrtp.//g///////..o.///// 09288.
Greta Tuckute, Nancy Kanwisher, and Evelina Fedorenko. Language in brains, minds, and machines. Annual Review of Neuroscience, 47, 2024a. URL https: / /www. annualreviews.
/tg/connenn/jourals///ee 1ttp:/...11/////eree/e 446/annurev-neuro 10eoo---2222231111- 22162222-1111422TAAA------
Greta Tuckute, Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. Nature Human Behaviour, 863)5444-561, 2024b. URL https: http///ww..ttttaaa com/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedingg.cer 21222434477e111ff00311111aa22222 fud053cccca4445--aape....
3f5ee243547dee91 fbd053c1
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Eygeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright,
17

---

Published as a conference paper at COLM 2025
Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay May- orov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimr- man, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. O Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. URL https: / http://dooooooo///////// wtt.://////...////.....
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhanc- ing vision-language model's perception of the world at any resolution. arXiv preprint 12191.
ar arXiv:2409.12191 2024. URL htttp:://ariiv. www.:/og///////ee/e/.ooo.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, An- thony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:: System Demon- strations, pp. 38-45. Association for Computational Linguistics, 2020. URL https: //www.cllebb org/ wwrgg/nnhollggyy//n emnlp-demos. 6.
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In International Conference on Learning Representations, 2025. URL https: ///penreview.net/ddf//o/e/ FideFFFAng...
Moritz F. Wurm and Alfonso Caramazza. Distinct roles of temporal and frontoparietal cortex in representing actions across vision and language. Nature Communications, 10(1): 289, 2019. URL https:/ /ttp:::///ww..eee wwwure. wtwm///tille//a/eeetoeooeeeeeeee $4 467-018- 1677-118-0808--
Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23)8619-8624, 2014. URL https:
/ http://wwwwpppg .nws. org/dor /doi//bs//// 1073/ 3/pass. 1403112111,
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. ar preprint ar drXim:2407.10671 2024a. URL https: //arxiv. org/ abs /22///2271112 10671.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar 2rXio:2412.15115, 2024b. URL https: / /ttp////.vvvv.////////e/////aav/ org/ /brg/2bsb/211.,. 151 15.
Shaoyun Yu, Chanyuan Gu, Kexin Huang, and Ping Li. Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension. Science advances, 10(21)eadd77444 7744, 2024. URL http://ooooooo/////////o//ooo org/ 10. 1126/ Sciadv.adn774.
18

---

Published as a conference paper at COLM 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36. Curran As- sociates, Inc., 2024. URL https::/proceeeeeeee 21ae93/f116622113333337722222230 4832 heee .p
neurips. cctpaper wiles//pper//0222
A
fMRI data
This section contains additional details on the fMRI dataset used in this study (Pereira et al., 2018, Experiment 1), summarized from the original publication, as well as description of our alternative processing choices. The stimuli and the fMRI data processed by Pereira et al.'s original processing pipeline are published linked at https: //osf. io/crwz7/.
Ability Accomplished
Cook
Food
Music
Sin
Counting
Garbage
Nation
Skin
Angry
Crazy
Gold
News
Smart
Apartment
Damage
Great
Noise
Smiling
Applause
Dance
Gun
Obligation
Solution
Argument
Dangerous
Hair
Pain
Soul
Argumentatively
Deceive
Help
Personality
Sound
Art
Dedication
Hurting
Philosophy
Spoke
Attitude
Del Deliberately
Ignorance
Picture
Star
Bag
Delivery
Illness
Pig
Student
Ball
Dessert
Impress
Plan
Stupid
Bar
Device
Invention
Plant
Successful
Bear
Dig
Investigation
Play
Sugar
Beat
Dinner
Invisible
Pleasure
Suspect
Bed
Disease
Job
Poor
Table
Beer
Dissolve
Jungle
Prison
Taste
Big
Disturb
Kindness
Professional
Team
Bird
King
Protection
Texture
Blood
Doctor
Lady
Quality
Time
Body
Dog
Land
Reaction
Tool
Brain
Dressing
Laugh
Read
Toy
Broken
Driver
Law
Relationship
Tree
Building
Economy
Left
Religious
Trial
Burn
Election
Level
Residence
Tried
Business
Electron
Liar
Road
Typical
Camera
Elegance
Light
Sad
Unaware
Carefully
Emotion
Magic
Science
Usable
Challenge
Emotionalle
Marriage
Seafood
Useless
Charity
Engine
Material
Sell
Vacation
Charming
Event
Mathematical
Sew
War
Clothes
Experiment
Mechanism
Sexy
Wash
Cockroach
Extremely
Medication
Shape
Weak
Code
Feeling
Money
Ship
Wear
Collection
Fight
Mountain
Show
Weather
Computer
Fish
Movement
Sign
Willingly
Construction
Flow
Movie
Silly
Word
Table 1: The 180 concept words of Pereira et al. (2018). Each word is a label of cluster of words obtained by performing spectral clustering over space of GloVe embeddings.
19

---

Published as a conference paper at COLM 2025
A.1 Concepts
The full list of 180 concepts used in the study is provided in Table Pereira et al. (2018) perform spectral clustering over a pre-trained English GloVe embedding space (Pennington et al., 2014) and manually label the obtained clusters. The list includes 128 nouns, 22 verbs, 29 adjectives and adverbs, and function word.
A.2 Stimuli
Figure 6 shows example stimuli for two of the 180 concepts under each paradigm.
The bird flew around cage,
Bird
The was just big enough for bird. The only bird she can see the parrot.
winged
nest
The bird poked its head out of the hatch.
The bird holds the worm in its beak.
mating
bird
beak
The bird preened itself for mating.
flock
Broken
The broken mirror went missing. The broken leg made the table unstable, dangerously teeters the
failing
bad
chair.
broken
The disorganized room is full of broken things. The computer was hopelessly broken. The door hung crooked after being broken.
croaked ruined
Figure 6: Example stimuli for two of the concepts in the Pereira et al. (2018) Experiment 1 dataset. The dataset includes six sentences, six images, and six spatial configurations of the same word cloud. The concept word is always bolded in the sentences and word clouds, and it is also added to every image in the picture paradigm. The participants were asked to think about the concept in relation the accompanying context or image.
A.3 Participants
It should be noted that the set of participants considered in this study (M01-M17) is not identical to that of Pereira et al. (2018): we exclude one participant scanned at Princeton (P01) but include the two novice subjects (M11, M12) that were excluded from the original analyses. The 17 participants (mean age 26.1, range 20-48; 10 men, 7 women; all native speakers of English; 14 right-handed, two left-handed, one ambidextrous) received payment for their participation, and gave informed consent in accordance with the requirements of the Committee on the Use of Humans as Experimental Subjects.
A.4 fMRI protocol
fMRI scanning was performed using a whole-body 3-Tesla Siemens Trio scanner with 32-channel head coil. Each two-hour scan session included 4-6 groups of 180 stimuli (one per concept), with each group randomly split into two runs (90+90 concepts). Each stimulus was presented for 3 seconds followed by 2-second fixation period, with additional 10- second fixation periods at the beginning, middle, and end of each run. The scan repetition time (TR) was set to 2 seconds.
A.5 fMRI data processing
The responses to each stimulus were estimated using general linear model (GLM) with additional denoising and regularization, implemented using the GLMsingle (Prince al., 2022) Python toolkit (version 0.0.1).5 Each stimulus presentation was modeled with boxcar function convolved with the canonical haemodynamic response (HRF). The time-series data is upsampled using PCHIP interpolation to TR=1s (from TR=2s in the original data from
20

---

Published as a conference paper at COLM 2025
Pereira et al.) in order to align the duration of the stimulus presentations (3s) with the TR boundaries. We set the following GLMsingle hyperparameters: number of GLMdenoise regressors fractional regularization level 0.05; default values for the rest.
B Semantic consistency ROIs
B.1 Statistically significant voxels
We determine the statistical significance of voxel's semantic consistency using independent permutation tests performed on two non-overlapping halves of the data.
First, we partition the stimuli set into two halves: for example, for given concept (e.g., Ability) and paradigm (e.g,, sentences), sentences 2, and 5 are allocated to the first half and sentences 3, 4, and 6 to the second half. Since for certain concept-paradigm pairs there are occasional participants who have only been presented stimuli out of the possible 6, we partition the stimuli in way that would result in the most even data split between the two halves: specifically, we choose split that minimizes the number of cases where a subject would have seen three stimuli from one half but only one from the other half.
We then perform a permutation test on the brain activations for each half of the stimuli.
For each paradigm Ω E {s, P, WC} we consider voxel's response vector in E IR 180 in which every element represents the strength of response to particular concept (computed based on the appropriate half of the stimuli only), always in the same order. We shuffle
the elements of in for each paradigm Q independently 1,000 times. Let P denote the vector resulting from the k-th shuffling (1 S k 1000); we can compute the "shuffled" 2(k)
correlation value ((k) by substituting Pn for pn for each paradigm in equation We
can then compute the one-sided p-value of the permutation test as 1000 T > C(K)),
where an indicator function.
Doing so for every voxel in every participant's brain yields two p-values for voxel. We select for each participant the set of voxels that were statistically significant on both halves of the stimuli (i.e., both p-values were below 0.05) and convert it to binarized 3D map (1 in statistically significant voxels, otherwise). Finally, we average the obtained binary map participants to obtain probabilistic map of semantically consistent voxels, where value corresponding to each voxel represents the percentage of participants in whose brain this voxel had statistically significant semantic consistency (Fig. 7).
right
30%
23%
15%
8%
0%
left
lateral
medial
Figure 7: Probabilistic map of voxels with statistically significant semantic consistency across all participants. Each voxel's value shows the % of the participants in whose brain this voxel has < 0.05 in both permutation tests.
21

---

Published as a conference paper at COLM 2025
ROI
Location
# voxels
Areas, named per Glasser al. (2016)
Auditory Complex (A5)
Area STSd posterior (STSdp)
ROI1 Superior temporal
975
Area Tenpono-Paricto-ociipttt Junction (TPOJ1)
Area TenporooPParieo-Occippa Junction 2
(TPOJ2)
Area IFSa (IFSa)
RO12
Inferior frontal
675
Area 45 (45) Area Frontal Opercular (FOP5)
Area TE2 posterior (TE2p)
ROT3 3
Ventral temporal
646
Area PH (PH)
Table 2:
The breakdown of the three identified left-hemisphere regions of interest (ROIs),
shown Figure 2c. The individual area definitions follow Glasser et al. (2016).
B.2
Defining ROIs
right
15%
10%
left
lateral
medial
Figure 8: The probabilistic map in Figure averaged by anatomical area as defined by Glasser et al. (2016). Thresholding these areas by probability, dividing the remaining ones into contiguous clusters, and filtering by size results the three ROIs shown in Figure 2c.
We use the probabilistic map in Figure 7 to define the boundaries of our regions of interest (ROIs). We use the anatomical parcellation of Glasser al., 2016 (in volumetric projection by Horn, 2016) and average the values of the probabilistic map over all voxels in each anatomical area. The result is shown Figure 8. After discarding all Glasser areas which the value is below 5.9% (i.e., 1/17, where 17 is the number of participants), we are left with 19 anatomical areas forming 10 contiguous regions of the brain cortex; of these regions lie in the left hemisphere. Finally, we filter these 10 regions by size (>600 voxels), which leaves the three left-hemisphere ROIs shown in Figure 2c. The size and anatomical makeup of each ROT is listed in Table 2.
B.3 Language response in semantic consistency ROIs
Mean language selectivity (measured per 55.5) for each ROI is reported in Table 3.
22

---

Published as a conference paper at COLM 2025
ROI
A Apsentenees,Noo-wordd
RO11
0.59 + 0.06
ROT 2
0.13 + 0.07
ROI 3
-0.66 0.04
Table 3: Mean (over voxels and participants) language selectivity in the semantic consis- tency ROIs. The value is measured as the effect size for the sentences non-words contrast of Fedorenko et al.'s 's (2010) language localizer. Standard error shown over participants.
C Models and methods
C.1 Sources and implementation
All aligmment experiments are implemented using the numpy (Harris et al., 2020), scipy (Virtanen et al., 2020), scikit-learn (Pedregosa al., 2011), and pandas (McKinney, 2010) libraries. For brain encoding, we use the RidgeCV class in scikit-learn to automatically tune the regularization hyperparameter via leave-one-out cross-validation.
We download all pretrained models from the HuggingFace Hub. Table 4 includes the links to the model repositories on HuggingFace. All experiments involving LMs are performed using Py PyTorch (Paszke al., 2019) and the transformers Python library (Wolf et al., 2020).
Model
HuggingFace ID
Parameters
GPT-2
openai-communityyggt22
124M
GPT-2 Medium
355M
GPT-2 Large
openai-coomuitty/gp2-lnrggg
774M
GPT-2 XI.
openai-communtty//gt22--1
1.6B
FLAVA
facebookk/flava-full
241M
Vicuna-1.5-7B
Lmsys/vicuna-7b-vi.55
7B
LLaVA-1.5-7B
1lave-hf/1lava-155--bbhff
7B
Qwen2.5-1.5B
Qwen/Qwen2.5-11.5
1.5B
Owen2.5-1.BBInntruue
Swen//Qwen2.511.5--mme
1.5B
Qwen2.5-3B
Qwen/Qwen2.5-38
Qwen2.5-3B-Instruuc
Swenn/Qwen2.5-BB-InSe
3B
Qwen2.5-VL-3B-Instte
Qwen/Qwen2.55VL
3B
Qwen2.5-7B
Twen/Qwen2.550
7B
Owenn.55-BBIInstruee
Twenn/Quenn25577-IInsrrruc 5-7B-Instrut
7B
Qween255-V---B-Insttuu
7B
Table 4: Pretrained language models used in this study. We provide the HuggingFace identifier and hyperlink for downloading each model's weights.
C.2 Stimuli input format
To input the stimuli from each paradigm (Fig. 6) into the models, we format them as follows:
Sentences are inputted as-is: The bird flew around the cage.
Word clouds are presented as sequence of space-separated words, with the concept word given first: bird nest flock mating beak winged. Since all word clouds for the same concept contain the same words, we use single sequence represent them all.
9ntpps://hugiingfaac.co
23

---

Published as a conference paper at COLM 2025
For picture + concept word inputs we add special VLM image tokens where needed: <image> Bird (LLaVA format) or visionn.sttrtt |>| image_pad v Bird (Qwen-VL format).
D Brain encoding performance
D.1 Whole-brain correlation with semantic consistency
Figure 3 in $6.1 shows how mean LM predictivity and semantic consistency correlate across anatomical areas. The semantic consistency of an area is measured by (1) obtaining the probabilistic map of reliably consistent voxels in that area across participants ($B.1, Fig. 7) and (2) averaging it over all voxels in an area (Fig. 8).
However, since main brain encoding experiment (Fig. 4) uses the raw value of C rather than the probabilistic map to group voxels, we rerun this analysis with C as the measure of semantic consistency. As can be expected, these two consistency measures are highly correlated (r = 0.83). The result is shown in Figure 9.
Sentence paradigm
Picture paradigm
Word cloud paradigm
Mean LM predictivty
Mean LM predictivity
Mean LM predictivity
Glasser areas
in ROI
in ROI 3 other
0.1
0.1
0.0
Semantic consistency
Semantic consistency
Semantic consistency
Figure 9: LM predictivity per Glasser al. (2016) area, with semantic consistency (x- axis) showing the mean value of the metric C. The Pearson correlations between C'add predictivity for each paradigm are: r(s] = 0.80, r[p] 0.22, r [Wc] = 0.75.
D.2 Inter-participant noise ceiling
In the whole-brain encoding experiment (Fig. 3), correlate LM predictivity in each anatomical area (defined per Glasser et al., 2016) with its level of semantic consistency. However, the higher predictivity might be not only due to increased brain-LM alignment: brain activations in some areas might be better predicted than in others because the signal there is simply less noisy. One standard approach the variability between the responses to the same stimulus in a given area of the same participant's brain (repeated trials). In our case, no participant sees the same stimulus more than once, i.e., trials are never repeated; instead, we compute the inter-pine pnteerpprticipant noise ceiling, estimating the variability responses to the same stimulus across all participants. We follow the procedure described by Tuckute et al. (2024b, section SI 5) to obtain an across-participant "noise ceiling".
estimate noise ceiling is to quantify
When we divide the mean LM predictivity in each area by its estimated noise ceil- ing, we find that the correlations with the area's semantic consistency (mean proba- bility of the area's voxels having statistically significant consistency) remain positive: ris] = r(4 = 0.02, r r|wc] = 0.63. Although other noise ceiling estimation approaches could offer additional insights (though they may not be feasible given the experimental design), these results confirm that our key findings hold even after accounting for cross- participant reliability.
24

---

Published as a conference paper at COLM 2025
D.3 Brain encoding across model layers
Figures 10 and 11 show how well the the participant-average brain activations in the chosen ROIs can be predicted from the representations extracted from different model layers (with mean and last-token pooling respectively). For each model, we choose the layer and the token pooling method that together yield the best performance, and use this setting for all other experiments in this paper. The middle layers are typically the most predictive, consistent with the observations of Caucheteux & King (2022) and Tuckute et al. (2024b).
The mean-pooled embedding layer (layer 0 in Fig. 10) serves as a baseline: it shows how well brain activations can be predicted from the non-contextual embeddings of the individual tokens. As expected, the predictivity at layer 0 is lower for sentences (where the syntactic information is important for reconstructing the meaning), but less so for word clouds or pictures (since layer 0 in VLMs includes the projected features from the vision encoder).
GPT-2
Qwen2.5-11
Qwen2.5-7B
FLAVA
GPT-2 Medium
Qwen2.55 5...5--111nstrrure
Qwen2 B-VL-3B-Instruct
GPT-2 Large
Qwen2.5-33
Vicuna-1.5-78
Swwen2.5-V--7B-Instie
GPT-2
Qwen2. 5-3--nrttrutt
LLaVA-1.5-78
ROI 1, Sentence paradigm
ROI 1, Picture paradigm
ROI Word cloud paradigm
0.7
0.6
Cross-validated
LM predictivity
0.0
ROI 2,
Sentence paradigm
ROI 2, Picture paradigm
ROI 2, Word cloud paradigm
0.7
0.5
Cross-validaled LM predictivity
0.5
0.4
0.3
0
10
20
30
40
50
0
10
20
30
40
50
0
10
20
30
40
50
ROI 3, Sentence paradigm
ROI 3, Picture paradigm
ROI 3, Word cloud paradigm
Crose-validatee
LM predictivity
4/14
0.4
b
0.3
0.2
0.1
0.0
10
30
40
0
30
40
0
30
40
Layer
Layer
Layer
Figure 10: Brain encoding performance by LM layer (with mean pooling over tokens). The target brain region (RŌI) activations are averaged over all participants ($5.3). The error bars show standard error of the mean over the five cross-validation folds.
25

---

Published as a conference paper at COLM 2025
GPT-2
Owen2.55 5.15B
Qwen2.5-77
FLAVA
GPT-2 Medium
Qwen2.55 5-1 1.5B-Instruct
Qwen2.5-7B-Insse
Qwen2. Swwen2.-VVL-3B--- 6-VL-3B-instruct
GPT-2 Large
Cwen2.5-3B
Vicuna-1.5-7B
Breen2.--VL-B--nssrree
GPT-2
Owen2. Cwen22553B-Inee 5-38-Instruct
LLaVA-1.5-78
ROI Sentence paradigm
ROI 1, Picture paradigm
ROI Word cloud paradigm
0.7
0.5
Cross-validaled LM predictivity
0.5
0.4
0
10
20
30
50
0
10
20
30
40
50
0
10
20
30
40
ROI 2, Sentence paradigm
ROI 2, Picture paradigm
ROI 2, Word cloud paradigm
Cross-valicatee
LM predictivity
0.5
0.4
0.3
0.2
10
30
40
0
10
30
40
0
10
30
40
ROI 3, Sentence paradigm
ROI 3, Picture paradigm
ROI 3, Word cloud paradigm
0.7
Cross-validaled
LM predicivitty
0.3
0.2
0.1
0.0
Layer
Layer
Layer
Figure 11: Brain encoding performance by LM layer (with last token pooling). The target brain region (ROI) activations are averaged over all participants ($5.3). The error bars show standard error of the mean over the five cross-validation folds.
D.4 Brain encoding by voxel quartile
Figure 12 shows the average (over models and participants) predictivity values per semantic consistency or language selectivity quartile (also shown in Figure 4), visualized as heatmap.
Figures 15-29 show how each LM's predictivity varies by language selectivity and semantic consistency quartile. Odd-numbered columns show how predictivity in each ROI changes across voxel quartiles by language selectivity, with each line corresponding to one semantic consistency quartile. Even-numbered columns show how predictivity changes across voxel quartiles by semantic consistency, with each line corresponding to one language selectivity quartile. The thickness of the line corresponds to the quartile (thicker-higher)) and the error intervals show standard error across participants. Each plot is averaged over participants; average over all models is shown in Figure 4. The correlations with both semantic consistency (C) and language selectivity (L) for each ROI are reported in the caption (averaged over paradigms).
26

---

Published as a conference paper at COLM 2025
ROI 1, Sentence paradigm
ROI 1, Picture paradigm
ROI 1. Word cloud paradigm
Quartlle by consistancy
0,073
0.082
0.12
0.14
Quarille by consistency
0.085
0.044
0.072
0.044
Sammiig by co coneagtanny
0.020
0.048
0.893
0.076
0.20
0.1
0.16
0.15
9
0.19
0.17
0.10
Quariin
0.005
0.15
0.19
0.2
0.23
0.54
0.15
0.15
0.16
0.2
0.23
0.25
0.27
0.00
Quartile by language
Quartile by language
Quartile by language
ROI 2, Sentence paradigm
ROI 2, Picture paradigm
ROI 2, Word cloud paradigm
Quartile by consistency
0.087
0.041
0.07
0.075
Quartiee consletency
0.351
0.65
0.063
0.071
0,027
0.0041
2.043
0.047
0.20
Quartlee by con
0.083
0.089
0.003
0.12
0.084
0.083
0.088
0.1
0.027
0.050
0.081
0.12
0.11
0.10
0.15
0.13
0.15
0.19
+
0.1
0.098
0.1
0.14
0.18
0.11
0.14
0.16
3
0.00
Quartile by language
Quartile by language
Quartile by language
ROI 3, Sentence paradigm
ROI 3, Picture paradigm
ROI 3, Word cloud paradigm
0.25
Quartlee by consistency
Quartil consintency
Monerro Sonmmn ba Socoonn
0.20
0.1
0.085
0.12
0.18
0.2
0.17
0:17
0.17
0.14
0.1
0.12
0.16
0.16
0:13
0:15
0.15
0.14
0.16
0.19
0.19
0.17
0.14
0.17
0.00
Quartile by language
Quartile by language
Quartile by language
Figure 12: LM predictivity by quartile for each ROI and paradigm, averaged over all models. The values shown reflect the mean correlation the ground-truth brain activations and the ones predicted from LM representations. In each heatmap, the x and y axes correspond to quartiles by language selectivity (sentences vs. non-words contrast; see $5.5) and by semantic consistency C respectively. Each cell of the grid shows the predictivity level on all voxels in a ROI that fall at the intersection of the given language selectivity and consistency quartiles. In ROI 1 (top row) and ROI 2 (middle row), the predictivity correlates with both language selectivity and semantic consistency, although the former is weaker for the word cloud (left column) and picture (middle column) paradigms. In ROT 3 (bottom row), only semantic consistency correlates with predictivity.
27

---

Published as a conference paper at COLM 2025
D.5 Brain encoding in the language network
For comparison with prior works on LM-brain alignment that target the brain's language network, we conduct additional analysis comparing the brain encoding performance in the left-hemisphere regions often engaged by linguistic processing (Fedorenko et al., 2010; Mahowald Fedorenko, 2016; Lipkin et al., 2022) with that in the semantic consistency ROIs identified in this paper. We use the six language parcels (located in the inferior frontal gyrus, orbital inferior frontal gyrus, middle frontal gyrus, anterior temporal lobe, posterior temporal lobe, and angular gyrus) created from a probabilistic overlap map from 220 participants. Since the semantic consistency ROIs are defined sets of Glasser et al. (2016) anatomical areas, we also identify all Glasser et al. (2016) areas that overlap substantially (by over 25% of an area's ss voxels) with any of the language parcels. If an area overlaps with more than one language parcel, we assign it to the parcel with the highest overlap. The brain encoding results are presented in Figure 13, reported by Glasser al. (2016) area.
RSA performance
To complement the analysis in $6.2, we conduct an additional RSA experiment using only the voxels with statistically significant semantic consistency ($B.1). This dramatically reduces the number of voxels to ~10-20 per ROI in most participants. The results (mean and SEM across participants) are shown in Figure 14. While the overall trends remain the same (see Figure 5), the VLM-ROI alignment gain from adding the picture paradigm data has decreased in the non-visual ROIs (1 and 2).
Downloaded from https://evlab. httpp//a..ssusspppp..eeoo/aaaaaa Laagus-ennggaggaaaaa SNaeee--N222mmmm. mii
28

---

Published as a conference paper at COLM 2025
Sentence paradigm
0.30
0.25
Mean LM prediclivity
STSdp_
STSdp_ TPOJ1
0.20
TPOJ1_.
STSA8==
A5
0.15
SYP=
IFSp_..
S STSVa_
PH_
0.10
471_
iF
TE2p-
0.05
TA2-
TPOJ2_
TPO13
TPOJ2_
FOPS_
0.00
IFGorb
IFG
MFG
AntTemp
PostTemp
AngG
ROI
Picture paradigm
TPOU2_
TPOJ2
0.25
TPOJ3..
Mean LM prodiclivity
PH
0.20
PHT TP
0.15
TPOJ1
TE2p-
0.10
M/=
#F
S1868-
ST ST5
STSdp_.
AEP
STGa_
0.05
A5
FOPS_.
TA2_
0.00
IFGorb
IFG
MFG
AntTemp
PostTemp
AngG
ROI
ROI
ROI
Word cloud paradigm
0.30
0.25
Mean predictivily
0.20
39 A6
0.15
A5_
45_
66b
$950e_
STEN2=
STSVP
PSL
TPOJ3..
PGs.
TE2:=
FOPS_
|FSa-
45
0.05
TP PGL
TPOJ2_
Pl
0.00
IFGorb
IFG
MFG
AntTemp
PostTemp
AngG
ROI
ROI
ROI
Figure 13: Brain encoding performance in the left-hemisphere language network parcels and the semantic consistency ROIs. For details, see Appendix D.5. LM predictivity is averaged over all models and participants. Each data point displayed corresponds anatomical area of Glasser et al. (2016). The language parcels from prior work are shown the left of the dashed line, and the semantic consistency ROIs are shown on the right. The Glasser et al. (2016) areas on the left are chosen by overlap (> 25%) with language parcels.
an
29

---

Published as a conference paper at COLM 2025
Language-only models
Vision-language models
Result (text images) only)
0.15
6.15
Pormuled bascline
Correlation
0.10
0.10
ROI
0.08
0.05
Result, (text images)
0.16
only)
0.15
2 Correlation
Permuled baseline
0.10
0.10
ROI
0.05
Result images) Result (text only)
0.15
0.15
Correlation
Permuted baseline
ROI
3
0.00
0.00
GPT-2
GPT-2 Medium
GPT-2 Large
GPT-2XL XL
Owen2 5-1 1.88
Owen2 5-1. 1.58 Instruct
Qwan2.5-38
Qwen2.5-55 5-38- Instruct
Qwen2.5-72 5-78
Qwen2.5-1 5-78- Instruct
Vicuna-1.500 15-78
FLAVA-
Qwen2 3B-instruct
Qwen2 5-V--
7B-Instruct
LLaVA-1.5-7B
Figure 14: RSA scores for each model when using only voxels with significant semantic consistency ($B.1). For more details, see Appendix and Figure 5. The three conditions correspond to using all three paradigms (text + images), using only sentences and word clouds (text only), and the baseline where the concepts are shuffled on one of the sides before computing correlations ($6.2). The results are consistent with those from using all voxels in each ROI (Figure 5), although the gains from adding images are reduced for non-visual ROIs 1 and 2. Error bars show standard error over participants.
Model: GPT-2
Sentence paradigm Controlled for
Word cloud paradigm Controlled language
language
consistency
consistency
Predishvity
ROI
30-
Prediciiity
022
ROI 2
Predicivity
ROI
80
QuartTe (cons.)
Quarttle (lang.)
Cuarttle (cons.)
Quartto (lang)
Figure 15: GPT-2 predictivity by voxel quartile.
= 0.0 0.24 0.07.
ROI 2: rc = 0.38 + 0.02, = 0.15 0.03. RO13: rC 0.27 - 0.02, =
30

---

Published as a conference paper at COLM 2025
Model: GPT-2 Medium
Sentence paradigm Controlled far
Word cloud paradigm Controllad for language
Cantralled for consistency
Controlled for consistency
language
Prediciiity
ROI
Prediciiviy Prediclivit
ROI à
ROI
Quartle (cons.)
Quartile (lang.)
Quartle (cons.)
Quartie (lang)
Figure 16: GPT-2 Medium predictivity by voxel quartile. ROI rc 0.42 = 0.03,
0.26 + 0.07. ROI 2: rc 0.36 1 0.04,rL 0.17 1 0.04. ROI 3: = 0.28 + 0.02, =
0.04 + 0.02.
Model: GPT-2 Large
Sentence paradigm
Word cloud paradigm Controlled language
Controlled
language
consistency
consistency
Predidivity
12
ROI
Predicivay
ROI 2
Predciiiity
ROI
Quartle (cone.)
Quartile (lang.)
Quartile (cons.)
Quartie (lang)
Figure 17: GPT-2 Large predictivity by voxel quartile. ROI 0.40 0.03, 0.29 +
0.05. ROT rc = 0.36 + 0.03, r = 0.14 + 0.02. ROI 3: rc = 0.27 + 0.03, = 0.04 + 0.02.
Model: GPT-2 XL
Sentence paradigm
Word cloud Controlled for language
Controlled for language
Cantrolled for consistency
Controlled for consistency
Predictvvty
022
ROI
Predisiiiity
82
ROI
60.
Predictiity
32
ROI l
i Quartile (lang.)
Quartte (cons.)
Quartila (cons.)
Quarlic (lang)
Figure 18: GPT-2 XL predictivity by voxel quartile. ROI 1: rc = 0.39 1: 0.02, = 0.23 1 0.05.
ROI 2: Fc 0.37 + 0.04, 0.16 i 0.04. ROI 3: rc 0.29 + 0.03, r 0.03 0.02.
31

---

Published as a conference paper at COLM 2025
Model: Vicuna-1.5-7B
Sentence paradigm Controlled far
Word cloud paradigm Controllad for language
Cantralled for consistency
Controlled for consistency
language
Predicevity
ROI
Predicliity
ROI 2
6.
Presicivviy
ROI
Quartle (cons.)
Quartile (lang.)
Quartiee (cons.)
Quartie (lang)
Figure 19: Vicuna-1.5-7B predictivity by voxel quartile. ROI FC 0.39 + 0.23 +
0.05. ROI 2: rc = 0.42 1 0.03, 0.16 i 0.04. 3: fc = 0.30 + 0.03, = 0.01 + 0.03.
Model: Qwen2.5-11 1.55B
Sentence paradigm
Word cloud paradigm for Controlled language
Controlled for language
Cantralled consistancy
conslstency
Prediolivity
12
ROI
Predicivity
ROI
2
6.
Predishiity
ROI
Quarlle (cons.)
Quartile (lang.)
Quartile (cons.)
Quartic (lang)
Figure 20: Qwen2.5-1.5B predictiiityy by voxel quartile.
0.31 +
0.05. ROI 2: rc = 0.36 + 0.05, r1 = 0.21 + 0.04. RO13: rc = 0.27 + 0.03, = 0.02
Model: Qwen2.5-1.5B-Imeeee
Sentence paradigm
Word cloud paradigm Controlled for language
Controlled for language
Controlled for consistency
Controlled for consistency
Predicavity
%0
ROI
30
Predidivity
ROI
Predicivity
ROI la
2 Quartile (cons.)
Quartle (oons.)
Quartile (jang.)
Quarttl (lang)
Figure 21: Qwenn..5-1.B-Instmmut predictivity by voxel quartile. ROI 1: Fc = 0.37 + 0.04, ri. 0.29 = 0.06. RO12: re 0.39 0.04, 0.19 + 0.04. ROI 3: re 0.26 0.03,
0.02 + 0.03.
32

---

Published as a conference paper at COLM 2025
Model: Qwen2.5-3B
Sentence paradigm Controlled for
Word cloud paradigm Controllad for language
Cantralled for consistency
Controlled for consistency
language
Predicivity
ROI
Predicliviy
02
ROI e
Presiciivtty
82-
ROI
Quartle (cons.)
Quartile (lang.)
Quartile (cons.)
Quartie (lang)
Figure 22: Qwen2.5-3B predictivity by voxel quartile. ROI 1: rc 0.39 + 0.02, F1,
0.25 +
0.06. ROI 2: rc 0.40 1 0.03, 0.18 i 0.04. ROI 3: fc = 0.29 + 0.02, FL = 0.04 0.02.
Model: Qwen2.5-3B-Insttuu
Sentence paradigm
Word cloud paradigm for Controllad language
Controlled for language
Contralled consistancy
consistancy
Prediolivity
02
ROI
Predictvity
02
ROI
2
Predistvity
ROI
3 Quartile (lang.)
Quarlle (cons.)
Quartile (cons.)
Quartic (lang)
Figure 23: Qwen2.5-3B-Instruct prediittiiit by voxel quartile. ROT rc 0.41 + 0.02, =
0.27 + 0.06. ROI 2: rc = 0.38 + 0.03,r1 0.20 + 0.04. ROI 3: rc = 0.28 + 0.02, rz
0.03 + 0.03.
Model: Qwen2.5-7B
Sentence paradigm
Word cloud Controlled for language
Controlled for language
Controlled for consistency
Controlled for consistency
Predichvvt
ROI
Prediciivity
02
ROI
80
Predictvity
ROI i
á Quartlle (lang.)
Quartte (cons.)
Quartilo (cons.)
Quartic (lang)
Figure 24: Qwen2.5-7B predictivity by voxel quartile. ROI 1: = 0.39 0.03, = 0.26 +
0.06. ROI 2: rc 0.39 + 0.03, /1 0.16 + 0.04. ROI 3: rc 0.27 1 0.03, 0.03
33

---

Published as a conference paper at COLM 2025
Model: Qwwnn2.5-7Biinsrrucct
Sentence paradigm Controlled consistency
Word cloud paradigm Controlled consistoncy
Controlled for language
Contrallad language
Predicliviy
ROI
0.
Predicliity
%
ROI
Predicivityy
92
ROI L
60
Quartle (cons.)
Quartile (lang.)
Quartile (cons.)
Quartie (lang)
Figure 25: Qwen2.5-78-Instruut predictivity by voxel quartile. ROI 1: rC 0.41 + 0.02, f1 = 0.23 i 0.05. ROI 2: rc = 0.36 1 0.05, = 0.21 i 0.03. ROI rc = 0.26 1 0.03, =
0.00 + 0.03.
Model: FLAVA
Sentence paradigm
Picture paradigm
Word cloud paradigm Controllad for language
Controlled for language
Controlled for consistency
Controlled for language
Controlled for consistency
Controlled for consistency
Prodictvity
ROI 1
2 Prodecivity
ROI
0.0
Predichilty
ROI
Quartlle (cons.)
Quartlle (ang))
Quartlle (cons.)
Quartile (lang.)
Quaitlle (cons.)
Quartile (lang.)
Figure 26: FLAVA predictivity by voxel quartile. ROT rc 0.34 r1
0.15 + 0.06.
RŌI 2: rc= = 0.32 + 0.04, ri = 0.15 + 0.02. ROI 3: rc = 0.24 + 0.02, rL = -0.00 + 0.03.
Model: LLaVA-1.5-7B
Sentence paradigm
Picture paradigm
Word cloud paradigm Controlled for language
Controlled for
Controlled for consistancy
Controlled for language
Controlied for consistency
Controlled for consistency
language
Predictivit
ROI
2 Prediciiity
RO12
Piedicivity
3.
ROI 3
60
8 Quartile (cons.)
2 Quartlic (lang.)
8 Quartila (cons.)
2 Quartile (lang.)
8 Quartlia (cons.)
2 Quartile (lang.)
= 0.14 + 0.06. ROT 2: rc = 0.35 + 0.04, r| = 0.14 + 0.03. ROI 3: rc = 0.28 + 0.03, r = -
Figure 27: VLa predictivity by voxel quartile. ROI rc = 0.35 r,
34

---

Published as a conference paper at COLM 2025
Model: Qwwen2.5VV--B--nnstruucd
Sentence Controlled for language
Picture paradigm
Word cloud Contralled for language
Controlled for corsislancy
Controlled for language
Controlled for corsislency
Controlled for consislancy
Predicivity
%
ROI
0.
ROI 2 Prodictulty
Prediciiityy
RDI3
Quarlile (cons.)
Quartiie (lang))
Quartile (cons.)
Quartile (lang.)
Quartile (cons.)
Quartile (lang.)
Figure 28: Owen2.5-VL-3B-Insteee predictivity by voxel quartile. ROI 1: rc 0.36 + 0.03, r1, 0.16 0.06. ROI2 2012:r: rc = 0.02, = 0.17 + 0.02. ROI rc 0.26 + 0.03, ri. =
-0.02 + 0.03.
Model: Qwwen2.5-VL-7--Insttuc
Sentence paradigm Controlled for
Picture paradigm Controlled for language
Word cloud paradigm Controlled for language
Controlled for consistency
Controlled for consistency
Controlled for consistency
language
Predichitt
ROI
a.
Predictvity
24
ROI 2
Preddiitt
22
ROI3
Quartile (cons.)
Quarttie (ang.)
Quartlle (cons.)
Quartile (ang.)
Quartlie (cons.)
Quartile (lang.)
Figure 29: Qwwnn..--L--B--nnstuutt predictivity by voxel quartile. ROI 1: rc 0.37 + 0.03, r1- = = 0.06. R012: = 0 = 0.19 + 0.02. ROI 3: rc 0.25 + 0.02, =
-0.02 + 0.03.
35


Reviews:
[{"review_id": "iNThiXkMXf", "reviewer": "Reviewer_ygPR", "summary": "This paper focuses on evaluating how well current language models align with human brain activity during naturalistic stimuli reading. The main objective is to identify the specific brain voxels that consistently respond to the same concepts, whether those concepts are shown in a picture, described in a sentence, or represented in a word cloud. To perform this, the authors consider both language-only models and vision-language models in their brain encoding experiments. The results show that the brain predictivity using language models have higher correlation in language-selective regions. Interestingly, visual regions show even stronger activity, even when using language-only models. The authors suggest that this is because language-only models can capture the meaning of concepts in a way that goes beyond just words, these models have the ability to capture cross-modal conceptual meaning.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors introduced a new metric to measure the semantic consisteny in the brain, and use it to find brain regions that represent concepts information consistently, irrespective of modality.\n2. The experimental setup is adequate\n3. The comparison between language-only and vision-language models yields interesting insights into how these models align with different brain regions.", "reasons_to_reject": "1. The major weakness of this work is that it neither introduces a novel scientific question in the study of brain-language model alignment, nor does it offer a new methodological approach for understanding how language models relate to brain activity. Specifically, the Pereiera2018 dataset, which the authors use, is already well-established in the Neuro-AI field, and has been widely used in previous brain encoding studies with both language-only [Schrimpf et al. 2021][Sun et al. 2022][Oota et al. 2022a][Aw et al. 2024] and vision-language models [Oota et al. 2022a]. Also, prior studies have already explored the use of multiple views to perform concept understanding in a context and whether the voxels are semantically consistent with language model representations across views [Oota et al. 2022b]. Therefore, this paper lacks a clearly defined research question and instead focuses on evaluating brain alignment across 15 different models without offering a strong, original scientific motivation.\n\n\n[Schrimpf et al. 2021] The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing, PNAS-2021\n\n[Oota et al. 2022] Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?, NAACL-2022\n\n[Oota et al. 2022a] Visio-Linguistic Brain Encoding, COLING-2022\n\n[Oota et al. 2022b] Multi-view and Cross-view Brain Encoding, COLING-2022\n\n[AW et al. 2022] Instruction-tuned LLMs with World Knowledge are More Aligned to the Human Brain, COLM-2024\n\n2. Given the nature of Periera2018 dataset, which is focused more on reading and understand concepts in different views, prior studies shown that even GPT-2 based models explain the variance really well [Schrimpf et al. 2021]. However, the authors use multimodal large language models like Qwen-2.5-7B and Qwen-2.5-VL for this dataset. Therefore, this raises questions about the motivation behind using such large-scale models for a dataset focused on relatively simple stimuli like short sentences and small images. \n\n3. The ROI selection criteria is not clear. Based on Figure 2, the authors focus on language regions including IFG, PTL and high-level visual cortex. However, considering Fedorenko's language-selective brain regions, areas like AG, ATL, IFGOrb, MFG, PCC and dmPFC. Additionally, prior studies have consistently shown strong brain-language model alignment across these broader language areas [Sun et al. 2019]. Without a clear criteria for the chosen ROIs, it's difficult to interpret the findings or compare them meaningfully to existing work.\n\n[Sun et al. 2022] Towards sentence-level brain decoding with distributed representations, AAAI-2019", "questions_to_authors": "1. The authors could explain how their definition of semantic consistency differs from the concept of informative, selective voxels as introduced in Pereira et al. (2018). The distinction between the two is not clearly addressed.\n2. It is unclear why the authors limit their evaluation of brain–language model alignment to only three ROIs.", "extra_scores": {}}, {"review_id": "PMlvcKUFCQ", "reviewer": "Reviewer_qokZ", "summary": "The work presents an analysis of brain-LLM representation alignment. The focus here is on concepts rather than words, where concepts can be represented either as text or visually. The study uses a dataset of fMRI readings collected in three settings (word in a sentence, word in a word cloud and word with accormpanying image). Based on Pearson's correlation analysis and RSA, the authors observe that certain areas of the brain consistently respond to textual and visual stimuli corresponding to the same concept, and furthermore, these responses can be reliably predicted from LLM and VLM representations, which is interpreted as evidence for a modality-agnostic encoding of concepts in LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written. \n\nThe novel method to identify brain regions of interest through a consistency analysis is well-motivated and seems reasonable.\n\nThe authors follow standard procedures in measuring brain-LM alignment and analyse the results in sufficient depth.\n\nA good number of older and newer LMs are included in the study. \n\nAn insightful finding is reported: LLMs and VLMs show stronger alignment to brain activations in the newly-identified consisency-based regions of interest, regardless of whether these regions play a role in linguisic processing, suggesting that these models encode a modality agnostic concept representation.", "reasons_to_reject": "My only concern is that the much of the work is dedicated to the novel method for region of interest identification, which in itself is not an NLP contribution. But since it is a prerequisite for the subsequent analysis, I reckon that's okay.", "questions_to_authors": "How is strong and weak response defined?\n\nWhat is the interpretation of a voxel that consistently responds to one concept (e.g. bird), but not another (e.g. art)?", "extra_scores": {}}, {"review_id": "Sy4cIYI2ah", "reviewer": "Reviewer_igwq", "summary": "The study compares brain imaging (fMRI) data from Pereira et al. (2018), which collected data from human participants exposed to multimodal conceptual stimuli, to representations from various small/medium-sized LMs and VLMs. This is measured by common metrics: brain encoding performance, i.e. predicting brain activations from model representations using ridge regression; and representational similarity analysis, i.e. spearman correlation between pairwise similarity matrices obtained from the each representation. These are correlated with a novel metric of semantic consistency as well as selectivity to language processing across brain regions.\nResults show correlation between semantic consistency and both predictivity and alignment.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Precise and clear overview of the fMRI data from Pereira et al. (2018), which is central to the experiments in this study. This is useful for the COLM community, which may be less familiar with human imaging data.\n\nExperiments are thorough and robust, employing various models, paradigms, modalities and controls.\n\nClear and focused contribution of the semantic consistency measure, which is explained clearly.", "reasons_to_reject": "Missing references about LM-brain alignment [1,2], particularly structural similarities [3], and about concept sharing in vision and language models [4].\n\nLacking discussion on what is known from neuroscience about the identified ROIs and whether the fact that they exhibit the highest semantic consistency is surprising or consistent with previous findings about the human brain.\n\n[1] Antonia Karamolegkou, Mostafa Abdou, and Anders Søgaard. 2023. Mapping Brains with Language Models: A Survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9748–9762, Toronto, Canada. Association for Computational Linguistics.\n\n[2] Gabriele Merlin and Mariya Toneva. 2024. Language models and brains align due to more than next-word prediction and word-level information. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18431–18454, Miami, Florida, USA. Association for Computational Linguistics.\n\n[3] Jiaang Li, Antonia Karamolegkou, Yova Kementchedjhieva, Mostafa Abdou, Anders S\\ogaard. 2024. Structural Similarities Between Language Models and Neural Response Measurements. Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations, PMLR 228:346-365, 2024.\n\n[4] Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders Søgaard. 2024. Do Vision and Language Models Share Concepts? A Vector Space Alignment Study. Transactions of the Association for Computational Linguistics, 12:1232–1249.", "questions_to_authors": "Since I am not perfectly familiar with the related literature, I cannot judge whether the semantic consistency metric is really novel or not. What similar metrics have been introduced before, and why are they insufficient? This is something that should be covered in related work. It just seems rather obvious. I am not saying it is not novel enough or too simple, just that I would imagine others to have introduced similar metrics before.", "extra_scores": {}}, {"review_id": "UYYoynpjF4", "reviewer": "Reviewer_tM7V", "summary": "This paper investigates the relationship between LLMs and brain activation during conceptual processing, using fMRI data from Pereira et al. (2018), including stimuli in three modalities (sentences, word clouds, and pictures). They define brain regions of interest (ROIs) based on this metric and examine model–brain alignment through both encoding models and representational similarity analysis. Results show that alignment is stronger in semantically consistent regions, even when these are not strongly language-selective.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper demonstrate an elegant way of repurposing the Pereira et al. (2018) multimodal dataset and introduce the semantic-consistency metric. \n2. Careful and creative ROI definition. It's nice to see that identified regions align with known semantic and visual processing areas.\n3. The use of both encoding and RSA methods provides converging evidence for the main claims.", "reasons_to_reject": "1. The title of the paper is not suitable for the claims listed in the paper. Even after going through the paper, I still don’t get it wha the role of language and concepts is in this context\n\n2. The observed increase in alignment between VLM and ventral temporal, it does not directly demonstrate that the models capture cross-modal conceptual meaning. To more convincingly support the idea that LLMs or VLMs capture cross-modal conceptual meaning, I would recommend adding a cross-modal encoding analysis—for example, training an encoding model using activations from the picture paradigm and testing it on responses to word clouds or sentences for the same concept. This would parallel recent work in multilingual encoding (e.g., de Varda et al., 2025), where cross-lingual generalization provided strong evidence for a shared meaning space.\n\n3. The paper lacks of inter-subject consistency analysis raising a serious concern to the robustness of the analysis which can potentially detriment the overall claim of the paper. The paper would be strengthened by incorporating ISC as an estimate of explainable variance in each brain region. As discussed by Bonnasse-Gahot & Pallier (2024), ISC can confound apparent regional differences in encoding accuracy. Including ISC would make the claims more grounded in neural signal quality. \n\n4. The paper does not include important details regarding layer-wise trends despite the use of best-performing layer per model and ROI especially because, unlike in LM Caucheteux (2022), the alignment behavior in VLMs are not well understood in this scenario.\n\nRef: \n- Caucheteux (2022) => Caucheteux C, King J R. Brains and algorithms partially converge in natural language processing[J]. Communications biology, 2022, 5(1): 134\n- Bonnasse-Gahot & Pallier (2024) => Bonnasse-Gahot L, Pallier C. fMRI predictors based on language models of increasing complexity recover brain left lateralization[C]//NeurIPS 2024-38th Conference on Neural Information Processing Systems. 2024.", "questions_to_authors": "1. Why the title is called \"The role of language and concepts in LM–brain alignment\". I am stilled puzzled after reading the whole paper. Could authors clarify what is the role specifically?\n\n2. It would also be informative to include baseline encoding analyses using simpler and cleaner features, such as word embeddings (e.g., GloVe) and visual features (e.g., CLIP). These baselines would help clarify how much of the observed brain predictivity can be attributed to conceptual abstraction versus modality-specific perceptual or lexical cues.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper looks at the alignment between human neural data and activations produced by LLMs and VLMs, emphasizing the presence of modality-agnostic brain regions which respond to both LLM and VLM representations. 

There was a lively discussion between the authors and reviewers, resulting in most reviewers being positive about the paper. There remains some hesitancy about the novelty, the clarity of the contribution, and the choice of metrics that are used. Because these objections are raised by the reviewers with the most expertise on the topic, they carry more weight than the overall average score. 

I believe that this paper could be published without any concern, but perhaps it would benefit more from another round of revision. During this revision, I would advise the authors to make clearer exactly what the research question is and how it relates to prior work, as well as motivate the methodological choices better (which, it appears, are their own contribution and perhaps should be highlighted as such).
Decision: Accept