fMRI predictors based on language models of increasing complexity recover brain left lateralization
Authors: Laurent Bonnasse-Gahot
Christophe Pallier
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
brain lateralization
neuroscience
language processing
fMRI
scaling laws
Primary area: neuroscience_and_cognitive_science
TL;DR: fMRI predictors based on language models of increasing complexity recover brain left lateralization for language, and the difference in brain score between left and right hemisphere follows a scaling law.

Abstract: Over the past decade, studies of naturalistic language processing where participants are scanned while listening to continuous text have flourished. Using word embeddings at first, then large language models, researchers have created encoding models to analyze the brain signals. Presenting these models with the same text as the participants allows to identify brain areas where there is a significant correlation between the functional magnetic resonance imaging (fMRI) time series and the ones predicted by the models' artificial neurons. One intriguing finding from these studies is that they have revealed highly symmetric bilateral activation patterns, somewhat at odds with the well-known left lateralization of language processing. Here, we report analyses of an fMRI dataset where we manipulate the complexity of large language models, testing 28 pretrained models from 8 different families, ranging from 124M to 14.2B parameters. First, we observe that the performance of models in predicting brain responses follows a scaling law, where the fit with brain activity increases linearly with the logarithm of the number of parameters of the model (and its performance on natural language processing tasks). Second, although this effect is present in both hemispheres, it is stronger in the left than in the right hemisphere. Specifically, the left-right difference in brain correlation follows a scaling law with the number of parameters. This finding reconciles computational analyses of brain activity using large language models with the classic observation from aphasic patients showing left hemisphere dominance for language.

fMRI predictors based on language models of increasing complexity recover brain left lateralization
Laurent Bonnasse-Gahot Centre d' Analyse et de Mathématique Sociales
Christophe Pallier
Cognitive Neuroimaging Unit
CNRS, EHESS 75006 Paris, France
CNRS, INSERM, CEA, Neurospin center
91191 Gif-sur- France
bbg@ehess.fr
christophe@ppllierr.rrr
Abstract
Over the past decade, studies of naturalistic language processing where partici- pants are scanned while listening to continuous text have flourished. Using word embeddings at first, then large language models, researchers have created encoding models to analyze the brain signals. Presenting these models with the same text as the participants allows to identify brain areas where there is significant correlation between the functional magnetic resonance imaging (fMRI) time series and the ones predicted by the models' artificial neurons. One intriguing finding from these studies is that they have revealed highly symmetric bilateral activation patterns, somewhat at odds with the well-known left lateralization of language processing. Here, we report analyses of an fMRI dataset where we manipulate the complexity of large language models, testing 28 pretrained models from 8 different families, ranging from 124M to 14.2B parameters. First, we observe that the performance of models in predicting brain responses follows scaling law, where the fit with brain activity increases linearly with the logarithm of the number of parameters the model (and its performance on natural language processing tasks). Second, although this effect is present in both hemispheres, it is stronger in the left than the right hemisphere. Specifically, the left-right difference in brain correlation follows scaling law with the number of parameters. This finding reconciles computational analyses brain activity using large language models with the classic observation from aphasic patients showing left hemisphere dominance for language.
Introduction
Since the seminal discovery that language disorders are most often associated with lesions to brain's s left hemisphere (Dax, 1865; Manning and Ahooas-AAnééion 2011; Broca, 1865; Wernicke, 1874), the existence of left-right asymmetry the cortical processing of language has been amply documented through different approaches, e.g., studies of split-brain patients (Gazzaniga and Sperry, 1967), intracarotid amobarbital injections (Wada and Rasmussen, 1960), electrocortical stimulation (Penfield and Roberts, 1959), functional brain imaging (Binder et al., 1996; Just et al., 1996; Stromswold et al., 1996; Malik-Moralede et al., 2022), and behavioral measurements such as reaction times to words presented in the left right visual fields (Hausmann et al., 2019). All in all, even there is clear evidence that the right hemisphere is implicated in speech and language processing (Bookheimer, 2002; lung-Beemann 2005; Lerner al., 2011; Vigneau al., 201 Bradshaw et al., 2017), it is estimated that left hemispheric dominance for language occurs in approximately 90% of healthy individuals (Josse and Mzoorio-Mazoyer, 2004; Tzourio-Mazoyer et al., 2017).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Given this state of affairs, one can only be surprised by the symmetric patterns highlighted by studies that have relied on computational models of language to predict fMRI brain time-courses in participants listening to naturalistic texts (e.g. Huth et al., 2016; Heer et al., 2017; Toneva and Wehbe, 2019; Caucheteux al., 2021; Schrimpf et al., 2021; Pasquiou et al., 2023). For example, Huth et al. (2016) constructed word embeddings using a latent-semantie approach based on co-occurrence counts (Landauer and Dumais, 1997) and used them as predictors of brain activity while participants listened to stories. The maps revealed by this approach were strikingly symmetric. This finding was replicated in subsequent studies that have used predictors derived from more advanced language models based on LSTM (Jain and Huth, 2018) Transformers (e.g. Toneva and Wehbe, 2019; Caucheteux et al., 2021; Schrimpf et al., 2021; Pasquiou et al., 2023) (but see Caucheteux and King, 2022 who reported significant left-right asymmetry). One potential interpretation is that brain scores are essentially driven by semantic representations (Kauf et al., 2024), supposedly represented in a very distributed fashion across both hemispheres (see e.g. the discussion in Huth al., 2016).
In this paper, we use 28 large language models (LLMs) of increasing size (from GPT-2 with 124 million parameters to @wwel.5-144 with 14.2 billion parameters; see Table A.1 for the full list) to fit IMRI data obtained from participants who listened to naturalistic text (from Le Petit Prince dataset, Li et al., 2022). We find that clear left-right asymmetry emerges as the size and performance of these models increases, and that this is not simply due to differences in signal-to-nowe ratio between left and right hemispheres.
2 Methods
2.1 fMRT data
Le Petit Prince dataset. We use the publicly available fMRI dataset Le Petit Prince' which provides recordings from English, French and Chinese participants who had listened to the audiobook of Le Petit Prince in their native language while being scanned using functional magnetic resonance (TR=2 s; voxel size=3.75 x 3.75 3.8 mm). Technical details on fMRI acquisition and preprocessing can be found in the publication accompanying the dataset (Li et al., 2022). Here, we use IMRI data spatially normalized in the Montreal Neurological Institute space, from all 49 English speakers. All of them were right- handed according to the Edinburgh handedness questionnaire. For each participant, fMRI acquisition was divided into 9 runs lasting each for about 10 min.
Additional preprocessing and creation of average subject. To reduce the computational burden of the study, and because we are interested in making inferences about the general population, compute a group average from all subjects. In order further reduce the computational cost of the study, all functional data are first resampled to 4 mm isotropic voxels, close to the original acquisition resolution. Before averaging, using nilearn compute compute_multi_____ _e_____p________ ppp__mak function with the threshold parameter set at 50%. The resulting mask, henceforth named whole brain volume, comprises 25870 voxels (1656 cm²). For all subjects and each run independently, voxels' time-series are high-pass filtered with a cut-off of 128 s. linearly detrended and standardized (zero mean and unit variance). We then compute the average subject by taking the mean of all these values per voxel and per run. Finally, we trim the first 20 s and the last 20 s of each run, as these were found to present deviation artifacts, and standardize the resulting sime-series
compute symmetric brain mask common all subjects
Inter-subjects reliable voxels. In order to evaluate the signal-to-noise ratio in each voxel, inde- pendently of any language model, we estimate the reliability of each voxel across participants. To do so, we split the group of all 49 subjects into two (almost) equal subgroups (24/25), compute the average response for each group and predict the BOLD time series from each voxel of one group from the activity of all the voxels from the other group. The fitting procedure follows the same method described in more details below when fitting brain response with neural network activations, and is based on a linear mapping from one set of responses to another, evaluated on held-out run through noss-validation, and regularized using ridge regression (L2). For each run, for each voxel, we thus compute the correlation between the true activity and the activity predicted from the other group of subjects, with a linear model trained on the other runs. This procedure repeated 10 times, using
2

---

b.
0.85
0.49
75%
20 0-2
04
86 28
Figure 1: Inter-subjects reliable voxels. (a) Distribution of inter-subjects correlations over voxels, computed using two subgroups of subjects and predicting one subgroup average fMRI time-course from the other one (see main text for details). Voxels with brain correlation above the dotted vertical line represent the 25% voxels with the largest correlations. (b) Glass brain representation of this inter-subjects reliability measure. Hot colors and dotted line show the 25% most reliable voxels.
different splits of the subjects. The final inter-group correlation for each voxel is the average over these 10 trials, and plotted Fig. 1. The resulting map is quite consistent to the ones obtained in previous inter-subjects correlations language studies (e.g. Lerner et al., 20.1). The plots presented in the main paper are based on the subset of the 25% most reliable voxels, representing total of 6468 voxels (414 cm²) 3297 in the left hemisphere and 3171 in the right hemisphere. These voxels are in hot-colored regions delineated with the dashed lines on Fig. 1. In Fig. B.I, inter-subjects correlations averaged over parcels defined by the harvard-Oxfort atlas are plotted. The graphics in panels (b) and (c) reveal a strong relationship between correlations in the left and right homologous regions, with a tendency toward stronger correlations in the left than in the homologous right regions.
Regions of Interest. We selected a set of a priori regions of interest from previous works on syntactic and semantic composition: the Temporal Pole (TP), anterior Superior Temporal Sulcus (aSTS) and posterior Superior Temporal Sulcus (pSTS) from Pallier et al. (2011), the Inferior Frontal Gyrus pars opercularis (BA44), triangularis (BA44) and orbitalis (BA47) from Zaccarella et al. (2017), and the Angular Syrus/Tempora Parietal Junction (AG/TPJ) from Price et al. (2015). Each region was defined sphere of 10 mm radius centered on the following coordinates in the Montreal Neurological Institute's s MNI152 space: : TP (-48, -27), aSTS (-54, -12, -12), PSTS (-51,-39, 3), AG/TPJ (-52, -56, 22), BA44 (-50, 12, 16), BA45 (-52, 28, 10), BA47 (-44.34.8).
2.2 Encoding models
Language models. We used 28 pretrained models, all available on the Hugging Face hub, ranging from 124M to 14.2B parameters. The full list along with some details the number of parameters, layers and hidden size (number of neurons Reeeurons at the output of each layer) is provided in Table A.1 on page 16 in Appendix. These pretrained models come from 8 different families, namely GPT-2 (Radford et al., 2019), OPT (Zhang et al., 2022), Llama 2 (Touvron et al., 2023), Qwen (Bai et al., 2023), gemma (Team et al., 2024), Stable LM (Bellagente al., 2024), Mistral (Jiang al., 2023), and Mamba (Gu and Dao, 2023). In this study, we only consider base models, trained with the same next-word prediction task. Most models are based the Transformer decoder architecture (Vaswani et al.. 2017), apart from the Mamba family, which is recent come-back of the recurrent neural network approach that is competitive with the Transformer language models.
In order to extract the activation of each model in response to the text of The Little Prince, each model is fed with the full original English text. We make use of the time-alle aligned speech segmentation provided in the Le Petit Prince openneuro.cmm org repository, to align the activity of the neural networks with what the subjects heard in the scanner. Each word the text is associated with its onset in the audiobook, and the representation that we compute is the average of all the vectors of all the tokens that constitute this word, as well as the following punctuation marks if there are any. Finally, in order to mimic the BOLD response, the activity of the artificial neural network under consideration is convolved with the Glover haemodynamic response function (Glover, 1999), as implemented in the nilearn package.
Baseline models. addition to considering all these pretrained language models, we look at a few baselines to compare with. First, we consider purely random vectors, aligned with the word onsets. Second, we consider random embeddings, similar to the previous case, but where each word is always associated with the same (random) vector. In these two cases, we look at 300 and 1024 dimensional
3

---

vectors. The results presented in this paper for these random baselines were averaged over 10 models obtained from different seeds. Third, we look at non-contextual embeddings provided by the seminal GloVe embeddings, where each word is always associated with the same vector, of 300 dimensions here, after training on co-oceurrences on large English text corpora (Pennington et al., 2014).
Procedure for fitting an encoding model. For each model, we separately consider the activity provided by each layer, plus the embedding layer. Hence, for I 12-layer model, we consider 13 layers. Let us notate X1,x the activity of a layer generated by the text of run k; (after convolution with the haemodynamie response function, described above), and Yo,e the BOLD time series of voxel v of run k. Xt, is a Rscans, & X nenurons matrix, and y, a vector of dimension Resaan, k. where s is the number of scans of run k. and Racurons the number of dimensions of layer / (for instance, gpt2-large model has 1280 dimensions, and Mistral-7B -B-v0. has 4096 dimensions). The goal is to predict the functional brain activity Ye, using the artificial neural activity X1,A as regressors, simply
using a linear mapping between the two: Yu, = x1,x3, where 3 is a vector of dimension necurons In order to avoid overfitting, the |3 coefficients are determined using ridge regression with regularization strength controlled by the parameter a. More precisely, given a run & used as test, all the remaining 8 runs are used training set determine the 3 coefficients. The hyperparameter is chosen using nested cross-validttion among a range of possible values (16 values log-spaced between 102 and 10"): among the runs used for training, one is used for validation, and the remaining runs for training the ridge regression with each value of a. Correlation between the actual values Y,, and the predicted
ones Ye is used decide which value of is used during training: the best a is chosen as the one yielding the greatest correlation averaged over all voxels 2, This value is then used to compute the ridge regression on the training runs and predict the functional brain activity on the test run k. The brain correlation for a given voxel v and given layer is given by repeating this procedure for all runs and taking the average correlation: 1/ runs Σisr cYrr(Y,,:: ; Y,,k). ). In the end, for a given voxel,
we take brain correlation the best correlation when considering all the layers of the model.
2.3 Computer code
The Python 3.10 code written for the present project relies on the following libraries: transforment learn v1 4.2 (Pedregosa et al., 2011), 1), nilearn vo. 10. 4, v4.40.1 (Wolf et al., 2020), 2019), matplotlib v3.8.0 (Hunter, 2007), seaborn 13.2 Pytorch v2.3.0 (Paszke et al.,
(Waskom, 2021), numpy v1. #1.26.4 (Van Der Walt al., 2011), pandas v2.2.2 (McKinney et al., 2010), statsmodels v0.14. 14.2 2 (Seabold and Perktold, 2010). All pretrained models were downloaded from Hugging Face through the transformers interface. The code is available at https:// /ttp:://ggttt...../ wtt..hcool..bg/////// waalllmmsinnnniaaeeea brain lateralization
The main English study on the average subject takes about five days of CPU time on a computer equipped with Intel Xeon w5-3425 processor (12 cores), mainly dedicated to perform the ridge regression of all layers of all models, using the scikit-les package.
3 Results
3.1 Distribution of correlations
The distributions of correlations over voxels, for each model, are displayed on panel (a) of Fig. 2 for the 25% most reliable voxels, and of Fig. B.2 for the whole brain volume. The random vector baseline yields very low brain correlations centered around O. Random embeddings (fixed random vector for each word) show positive correlations, especially in the 25% most reliable voxels mask, and increases with the dimension of the embeddings from 300 to 1024. Glo word embeddings perform better than these baselines. Finally, the contextual embeddings provided by the large language models yield even higher brain correlations.
"We performed exploratory analyses showing that a single value for a for all the voxels in the brain was not detrimental the predictions of the functional data, and produced massive gain speed.

---

5.66
ardam vestors 30243
1r enorodings ngs 3030
lrddom anouod nga 20246
Givicc
lxnge motele
8.43
3.42
240
327 number parameters
brain correlation
Gemma
StableLM
Mistral
Mamba
correlatoon
0.45
0.44
0.43
0.42
0.41
10a
10aº10"
10a
10
10
10ºº10a
10a010"
100010"
10a 102010" 10 10a
10
number parameters
Figure 2: Performance of various models in predicting IMRI brain time-courses. (a) Density estimates of the distributions of r-scores obtained for all 28 large language models, the random baselines and GloVe. The densities are scaled to have the same maximum (b) Average r-score as a function of the number of parameters of the model, in log scale. Here and in the next figures, the shaded area indicates the 95% confidence interval of the slope, computed with bootstrap. (c) Same, split by models' family.
3.2 Scaling law of fMRI encoding models
Focusing on the contextual large language models, Fig. 2b shows the relationship between the size of the language models and their prediction performance, that is, the average correlation across the 25% most reliable voxels. The linear correlation between the brain correlation and the logarithm of the number of parameters is 0.95, 7.3e 15. The corresponding plot for the whole volume, displayed in panel (b) of Fig. B.2, also shows a linear relationship (r 0.95, 3.2e 14). This finding replicates the scaling law first described by Antonello al. (2024).
Fig. 2c details the results by the family of models. Older models like the ones from the OPT family perform worse than more recent ones like Mistral Qwen, although within the same family the scaling law holds well in general, as exemplified by the GPT-2, OPT or Qwenl.5 families. Interestingly, the Mamba family, which is based a recurrent neural network architecture contrary to all the other, Transformer-basee of the envelope of the results (see Fig. B.3). In addition, we performed an analysis of the fit function of layer depth, depicted in Fig. B.4. The results confirm earlier reports by Toneva and Wehbe (2019) and Caucheteux and King (2022) that the middle layers are the most predictive ones.
models, has similar brain scores, although they lie at the bottom
Brain maps of smallest vs. largest models. Fig. with the two most extreme models: GPT-2 with the smallest number of parameters and Qwenl 1.- 1.5--4B with the largest number of parameters. Interestingly, while the map generated by GPT-2 is quite symmetric, Qwen 1.5-14B fits fMRI responses better in the left hemisphere than in the right (while overall performing better than GPT-2).
shows the brain correlations maps associated
Voxel-wise sensitivity to model size. To investigate the strength of the relationship between model size and brain score in each voxel, we compute the slope of the linear regression line between the logarithm of the number parameters and correlation score. These slopes are displayed Fig. The strongest effects of model size are detected in the left angular gyrus, the medial prefrontal cortex and the precuneus on the medial surface of the parietal lobe. Smaller effects are detected in the middle temporal and inferior frontal gyri.
5

---

a.
GPT-2 (124M parameters)
b.
Qwen1.5-14B (14.28 parameters)
0.00 0.11 0.22 0.33
0.44
0.00 0.11 0.22 0.33
0.44
Figure 3: Brain correlation maps associated with the smallest (a) and the largest model (b). These maps show the increase in r-score relative the model using the random embedding baseline 1024d.
0.032
0.016
0.008
0.000
Figure 4: Voxel-wise strength of the relationship between models' size and their predictive power. The slopes of the linear regression between r-score and the logarithm of the number of parameters are presented on glass brain view. For readability, only voxels with p-values smaller than 10-7 are shown.
a.
b.
0.45
2.03
0.43
brain
0.42
2.42
0.43
0.42
15 perplesity
hellawwing
0.460
0:30
0.86
brain correlation
atton
pa.4.ee-00 01
1.5e 03
3.00
03
0.450
0.445
0,440
1x
B 101
10
70 75
85
parameters
perplexity
hellaswag
Figure 5: Brain correlation and performance natural language tasks. (a) Brain correlation on the 25% most reliable voxels for all 28 large language models as of a function of perplexity on Wikitext-2 test Note that the x-axis is inverted, the lower the perplexity the better the model. (b) Same with performance on the Hellaswag benchmark. The higher the better. (c) Same as Fig. 2b and (a) and (b), but focusing on the 10 largest models, with number of parameters above 3B. See Fig. B.5 for similar plots on the whole brain volume.

---

0.77
b. 0.020
0=1.ee11
0.45
brain correlation: -R
0.015
0.44
0.010
0.43
0.42
0.40
0.005
10'
10a
10th
number parameters
number of parameters
Figure Emergence of left lateralization with the size of the encoding models. (a) Brain correlations on the 25% most reliable voxels as function of the number of parameters, for the left hemisphere (red) and for the right hemisphere (green). (b) The difference between left and right hemispheric brain correlations also shows scaling law.
Brain correlation and performance on natural language tasks. The number of parameters of model is rough proxy of its performance on language tasks. For example, OPT-13B is found to have lower performances on various natural language processing tasks than Mistral-7B which has almost half the number of parameters. Hence, we also look at measures of models' performance beyond their raw number parameters, First we compute the perplexity of all the models on the Wikitext-2 test set (Merity et al., 2016). Second we look at the Hellaswag benchmark (Zellers et al., 2019), which aims to measure the ability of model to understand language. This measure was found be quite challenging for language models, and the performance of models scale on this benchmark well with the training budget, either in terms of number of parameters or number of tokens seen during training (see Touvron et al., 2023). Fig. 5 shows how brain correlation increases with better models, either as measured by lower perplexity on the Wikitext-2 test set or on the Hellaswag benchmark. Both measures exhibit strong correlation with brain fit (r -0.91, p 1.le 11 for perplexity and r = 0.95, = 4.2e 15 for Hellaswag). But this is particularly relevant when looking at the ten largest models, with number of parameters above 3 billions: whereas the correlation between brain score and number of parameters is no longer significant in this case (r = 0.30, = 0.41), the other measures of performance display significant correlation and r 0.83,p p = 3.0e - 3 for Hellaswag): see Fig. 5.
0.86, p 1.5e 3 for perplexity
3.3 Left-right hemispheric asymmetry
Voxel-based analysis. The maps presented Fig, 3 and Fig, 4 hint at left-right differences. Therefore, we plot the relationships between model size and brain scores split by hemisphere in Fig. 6a. This graphic reveals that the larger the model, the stronger the left-right asymmetry. More precisely, while the smallest models have similar r-scores both in the left and right hemisphere, the largest models yield stronger r-scores in the left one. The interaction between model size and hemisphere can be assessed by the correlation between model size and the left-right difference in r-scores, displayed in Fig. 6b. This graph shows that the difference also follows scaling law (r = 0.89, p = 1.8e-1i -10). See Fig. B.6 for a similar analysis on the whole brain volume. Finally, we did for the brain scores (Fig. B.4), examined the effect of layer depth on the asymmetry. This analysis, reported on Fig. B.7, shows that the layers with the stronger fit also have the strongest left-right asymmetry.
Regions of Interest analysis. The relationships between model size and r-scores in seven regions of interest from the language network in the left hemisphere, and in their mirror-image regions the right hemisphere, are shown Fig. 7. Brain scores improve with model size in all regions, both in the left and in the right hemisphere. Except for pars opereularis (BA44) in the inferior frontal gyrus, and for the temporal pole (TP), a left-right asymmetry in favor of the left emerges when model size increases. The steepest effect occurs in the Angular gyrus/ camporoo-paietta. This region, AG_TPJ, was already highlighted in Fig. 4 showing the slopes in individual voxels.
7

---

b.
0.020
0.015
0.010
0.005
TP
5STS
PSTS
AG AF)
0.000
3STS
PSTS
AS TPI
B445
C.
aSTS (*)
PSTS (M)))
AG. TP) (m0))
BA45 (oar)
BA47 (*))
over random ambedainn
0.35
brain correlatioo
0.25
0.15
10a
10a
1002
30*
1.0
1.041
10*
1.0
102
101
10"
1044 10*
10*
1.02
10*
1.0
1.022
10*
100
1032
number of
Figure 7: Model size impact in various Regions of Interest. (a) Locations of ROIs. (b) Slope in r-score as a function of the logarithm the number of parameters, averaged over all voxels of given ROL Error bars indicate 95% confidence interval, estimated by bootstrap. The annotation below each ROI label indicates an estimate the level statistical significance of the difference between the slopes of the voxels of left ROI compared to its respective right ROI (ns:
significant, *:p < 0.05, **:p < 0.01, *** < 0.001). (c) Brain correlations over the random embedding baseline (see Fig. B.9 page 22 for similar plots with raw brain correlations),
function of the logarithm of the number of parameters. The annotation next to each ROI label indicates the level of statistical significance of the correlation between the difference r-scores of the models on the left hemisphere minus the right hemisphere, as a function of the logarithm of the number of parameters, testing the interaction between hemisphere and model size. See Table A.2 for full statistics.
Relationship between signal-to-nois ratio, brain scores, and growth in asymmetry. Do large models fit the left hemisphere better than the right simply because the signal-to-noise ratio, ceiling of explainable signal, is higher in the left voxels than in the right ones? A proxy for the explainable signal is the model-free inter-subject correlation (ISC). Fig. B.1 shows that the ISC indeed tends to be stronger on the left. To determine this could explain our findings, we conducted several analyses presented on Fig. B.8 and Fig. B.9. First, computed "normalized r-scores" by dividing raw r-scores by the ISC in each voxel. Panels (a) and (b) Fig. B.8 display the relationship between these normalized brain correlations and model size. The left-right asymmetry previously observed with raw r-scores on Fig. still holds. Next, using the parcels from the Harvard-Oxford atlas, we plot the relationship between ISC, brain correlations and their asymmetry (Fig. B.8. bottom panels). Although ISC and brain correlation correlate (panel c), the left-right difference in slope of brain correlation as function of model size does not covary with the interhemispheric difference in ISC (panel d). Finally, we computed the ISC in our Regions of Interest. Fig. B.9 suggests that while differences in ISC partly explain the average brain score difference, they do not explain the increase in difference between left and right hemispheres, across models.
Impact of model training. In order assess the impact of model training on the left-right asymmetry, we compared r-scores for randomly initialized, untrained models, and for trained models, from the GPT-2 and the Qwen1.5 families. Fig. 1. shows that, with training, brain correlations get better on the left hemisphere compared to the right one. Moreover, while the left-right score difference increases with the number parameters for trained models, this relationship breaks down and is flat in the case of the untrained models. Another way to assess the influence of model training is fit the same L.L at various stages of training. The Pythia family (Biderman et al., 2023) provides many intermediate checkpoints during training, from the untrained model, randomly initialized,

---

to the full model, trained on one epoch on the training set of 300B tokens from the Pile dataset. Here we consider Pythia-6.9B, with 32 layers and 4096 dimensions. Fig. B.10b shows the impact of training on brain correlations. The model's fit improves as training proceeds and the left-right asymmetry, originality biased towards the right hemisphere, increases, reaching a plateau favoring the left hemisphere.
3.4 Extension other languages
As left dominance is universal (see e.g. Malik-Moraleda et al., 2022; it is even attested in sign language, Poizner et al., 1987), we wanted to check if our result holds for other languages. We therefore applied our approach to Chinese and French data provided in Le Little Prince dataset (considering all 35 Chinese participants and all 28 French participants).
We first computed inter-subjects correlations. Their distributions and the corresponding maps are displayed on Fig.B.11 and Fig.B. B.12 panels (a) and (b). The correlation values are lower than for English (Fig. B. I), but the most reliable voxels encompass the same brain regions. Panels (d) and (e) reporting correlations in regions from the Harvard-Oxfone atlas (shown in panel (c)) and their left- right difference qualitatively replicate the previous observations in English results, that is, left-right asymmetries are detected in the same regions.
Although much fewer LLMs are available than for English, some models on the Hugging Face hub were trained on Chinese or French data. We considered 10 models for Chinese, ranging from 12M to 14B parameters, and 4 models for French, ranging from 124M to 7B parameters (see panel (f)). The relationships between model size and r-scores are presented in panels (g), (h) and (i) of Fig. B.11 and Fig. 1.12. They confirm the scaling law, and the emergence of the left-right hemispheric asymmetry two languages different from English.
3.5 Individual analyses
The analyses presented above were performed on data averaged across individuals. It is important to verify if the emergence of asymmetry holds at the individual level or is an artifact of averaging. Because of limitations in time and computational power available to us, we could only analyze five participants subset of models. To avoid any bias, selected the first 5 English participants. The results, presented in Fig. B.13, show that three of them present a significant increase in asymmetry (p 0.05) when considering the full brain (panel c), and four when considering the 10% voxels with the highest r-scores in each hemisphere (panel e).
4 Discussion & Conclusion
By manipulating the size of artificial neural language models used to fit fMRI data of naturalistic language comprchension, we observed: (1) linear relationship between the logarithm of the number of parameters of models and their ability to predict the IMRI time-courses (Fig. 2) and (2) left-right asymmetry emerging when encoding models are based on increasingly complex models (Fig. 6).
The first observation replicates the finding of Antonello et al. (2024) who described a scaling law between model size and brain score, extending to brain data the scaling laws observed between model size and performance on various NLP tasks (Kaplan et al., 2020). Here, we generalize their result to a different dataset that includes three languages, and using larger variety of models and families that notably include non- TranTroffffrrr Mamba.
The scaling law breaks down when focusing on the largest models, above 3B parameters. Yet, in this range, indices assessing the performance of the model on NLP tasks are still predictive of brain score (Fig. 5). The relationship between neural network models performance in NLP tasks and brain scores, was also investigated by Schrimpf et al. (2021) and by Caucheteux and King (2022). Both teams reported that brain scores generally improved as the performance in the next word prediction task increased, extending to the field of language processing results that were found in the visual domain by Schrimpf et al. (2020) or in the auditory domain by Kell et al. (2018). Caucheteux and King (2022) found that brain scores reached plateau, and even dropped somewhat (see their Fig.2g) which might be due to an overspecialization for the next word prediction task in their study (Caucheteux et al., 2023).

---

Our second observation is the emergence of an asymmetry with increasing model size, finding that has not been reported in the literature $. A quick but vague explanation is that larger models better capture language representations in the left hemisphere. It begs the question of which representations are improved in larger models relative to smaller ones. For example, it lexical, syntactic, semantic, pragmatic or world knowledge? The present work calls for detailed comparisons of the features discovered by large vs small language models, and how and where this translates into better IMRI prediction.
One can attempt to perform inverse inference, that is, try to infer which processes are involved from the brain areas highlighted by our analyses, keeping in mind the pitfalls of such an approach (Poldrack, 2011). The cortical areas where model size has the strongest impact are the Angular gyrus, the Precuneus and the medial prefrontal cortex (mPFC). These regions are also those where the effects of training a (small) network are the strongest Pasquiou et al. (2022) and were fit by models trained on semantic but not on syntax by Pasquiou et al. (2023). These regions are not specific to language, e.g. they are part of the default mode network, but are involved the highest levels of language comprehension (Simony et al., 2016; Chang et al., 2022). The mPFC and Precuneus are known to be sensitive to discourse coherence (Ferstl and von Cramon, 2001; Xu et al., 2005). The Angular Gyrus is also considered part of the semantic system (Seghier et al., 2010; Binder and Desai, 2011 I; Price et al., 2015; Kuhnke et al., 2023). This suggests that the main effect of increasing model size is to improve the model capabilities at the semantic and pragmatic levels. The analysis by regions of interest revealed that the asymmetry holds in most regions of the core language network, except pars opercularis (BA44) and the Temporal Pole. The left BA44 has been associated with syntax (Zaccarella and Friederici, 2016) and/or speech processing (Matchin and Hickok, 2020). The temporal poles, both on the left and on the right, are linked to semantic processing (Pobric et al., 2010; Pylkkänen, 2019) and have been designated as an amodal semantic hub by Patterson and Lambon Ralph (2016) (but see Snowden al., 2018).
is remarkable that in all regions of interest, even in the right hemisphere, the largest models' fits always improve over the smallest models (see Fig.7). It has been known for long that the right hemisphere has some language capacity (Bradshaw et al., 2017), which can manifest in split brain patients or in patients with lesions in the left hemisphere (Vigneau et al., 2011). The right hemisphere has been associated with speech processing, especially prosody, usually considered to be processed in the right temporal lobe (Wildgruber et al., 2006), as well as with high-level aspects of language understanding involved in the comprehension of metaphors or jokes Jung-Beeman 2005; Bookheimer, 2002). More work will be needed to determine which aspects of LLMs allow them to fit these regions.
Our main analyses were performed on fMRI data averaged across participants because of com- putational power limitations. We report preliminary analyses on 5 participants showing that the emergence of asymmetry holds at the individual level (see Fig. 13). Nevertheless there is an im- portant variability between subjects in brain correlations, in the difference between left hemispheric vs. right hemispheric average brain correlations, and in the slope of the left-right difference. It would be interesting for future work to assess the reliability of these measures, and to compare them with independent assessments of individual language hemispheric dominance. Unfortunately, such measurements are not provided in the fMRI dataset that we use.
An interesting question concerns the evolution of lateralization during the training process of large language models, either by training such a model from scratch or by having access to checkpoints during the training (see Antonello et al., 2024; Pasquiou et al., 2022, for first attempts). In humans, it seems that language processing in the brain starts bilaterally and then becomes progressively lateralized over time (Szaflarski et al., 2006; Olulade et al., 2020:
Paechiv et al., 2024), although some studies have found very carly lateralization (Witelson and Pallic, 1973; Wood et al., 2004). Our preliminary exploration with the Pythia-6.9B model (Fig. B.10) reveals that the left- right asymmetry emerges during the training process. This leaves open the questions of which representations improved by learning are responsible for this asymmetry, and whether parallels with the development of language acquisition can be drawn.
'Note that the growing asymmetry with model size goes beyond showing left- difference in a single model, which could be explained by signal to noise ratio asymmetry (see, e.g., Caucheteux and King, 2022, Fig. 2d)
10

---

Acknowledgments
We would like to sincerely thank all four anonymous reviewers, whose constructive feedback signifi- cantly contributed to the improvement of the paper. Many thanks also to Yair Lakretz, Emmanuel Chemla, and all the participants of the Linguae ML seminar at École Normale Supérieure.
References
Antonello, R., Vaidya, A., and Huth, A. (2024). Scaling laws for language encoding models in fMRI.
Advances in Neural Information Processing Systems, 36.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F,, et al. (2023). Qwen technical report. arXiv preprint arXiv:2309.166009
Bellagente, M., Tow, J., Mahan, D., Phung. D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., et al. (2024). Stable Im 2 1.6 technical report. arXiv preprint arXiv:2402.17834.
Biderman, S., Schoelkopf. H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023). Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR.
Binder, J. R. and Desai, R. H. (2011). The neurobiology of semantic memory. Trends in Cognitive Sciences, 15(1 15:75555.
Binder, R., Swanson, S. J., Hammeke, T. A., Morris, G. L., Mueller, W. M., Fischer, M., Benbadis, S., Frost, J. A., Rao, S. M., and Haughton, V. M. (1996). Determination of language dominance using functional MRI: a comparison with the Wada test. Neurology, 46(4):978-99)
Bookheimer, S. (2002). Functional MRI of Language: New Approaches to Understanding the Cortical Organization of Semantic Processing. Annual Review of Neuroscience, 25(1):151-188
Bradshaw, A. R., Thompson, A., Wilson, A. C., Bishop, V., and Woodhead, Z. V. (2017). Measuring language lateralisation with different language tasks: a systematic review. PeerJ, 5:e3929.
Broca, P. (1865). Sur le siège de faculté du langage articulé. Bulletins et Mémoires de la Société d' d'Anthropologie de Paris, 6(1):377-393.
Caucheteux, C., Gramfort, A., and King, J.-R. (2021). Disentangling syntax and semantics in the brain with deep networks. International conference on machine learning, pages 1336-1348. PMLR.
Caucheteux, C., Gramfort, A., and King, J.-R. (2023). Evidence of predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3):430-441.
Caucheteux, C. and King, J.RR. (2022). Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):1-10.
Chang, C. H. C., Nastase, S. A., and Hasson, U. (2022). Information flow across the cortical timescale hierarchy during narrative construction. Proceedings of the National Academy of Sciences, (19(1))e22093307 19. Publisher: Proceedings of the National Academy of Sciences.
Dax, M. D. (1865). Lésions de la moitié gauche de l'encéphale coincidant avec l'oubli des signes de la pensée: Lu Congrès méridional tenu à Montpellier 1836, par le docteur Marc Dax. Gazette Hebdoomadaire de Médecine et de Chirurgie, 17:259-260.
Ferstl, E. C. and von Cramon, D. Y. (2001). The role of coherence and cohesion in text comprehension: an event-related fMRI study. Cognitive Brain Research, 11(3):325-343
Gazzaniga, M. S. and Sperry, R. W. (1967). Language after section of the cerebral commissures. Brain, 90(1):131-1)) Publisher: Oxford University Press.

---

Glover, G. H. (1999). Deconvolution of impulse response in event-related BOLD fMRL. Neuroinage, 94):416-429
Gu, A. and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.
Hausmann, M., Brysbaert, M., van der Haegen, L., Lewald, J., Specht, K., Hirnstein, M., Willemin, J., Barton, J., Buchilly, D., Chmetz, F., Roch, M., Brederoo, S., Dael, N., and Mohr, C. (2019). Language lateralisation measured across linguistic and national boundaries. Cortex, 111:134-144 147.
Heer, W. A. d., Huth, A. G., Griffiths, T. L., Gallant, J. L., and Theunissen, F. E. (2017). The Hierar- chical Cortical Organization of Human Speech Processing. Journal of Neuroscience, ( 37(27):6539- 6557. Publisher: Society for Neuroscience Section: Research Articles.
Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 933:900-95.
Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, E., and Gallant, J. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-44 458
Jain, S. and Huth, A. (2018). Incorporating context into language encoding models for fmri. Advances in neural information processing systems, 31.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., Casas, D. d. 1., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023). Mistral 7B. arXiv preprint 1rXiv:231000682.
Josse, G. and Tzourio-Mann Mazoyer, N. (2004). Hemispheric specialization for language. Brain Research Reviews, 44(1):1-12
Jung-Ben Lung-Beeman, M. (2005). Bilateral brain processes for comprehending natural language. Trends in Cognitive Sciences, 911):512-518.
Just, M. A., Carpenter, P. A., Keller. T. A., Eddy, W. F., and Thulborn, K. R. (1996). Brain activation modulated by sentence comprehension. Science, 77455844):144-11
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.0836.
Kauf, C., Tuckute, G., Levy, R., Andreas, J., and Fedorenko, E. (2024).
Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in t he Language Network. Neurobiology of Language, 5(1):7-42.
Kell, A. J., Yamins, D. I.., Shook, F. N., Norman-Haignere S. V., and McDermott, H. (2018). task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-644.
Kuhnke, P., Chapman, C. A., Cheung, V. K. M., Turker, S., Graessner, A., Martin, S., Williams, K. A.. and Hartwigsen, G. (2023). The role of the angular gyrus in semantic cognition: synthesis of five functional neuroimaging studies. Brain Structure and Function, 228 (28(2):733-21
Landauer, T. K. and Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychologicul review, 104(2):211. Publisher: American Psychological Association.
Lemnrr, Y., Honey, J., Silbert, J., and Hasson, U. (2011). Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using Narrated Story. Journal of Neuroscience, 3188:2200622 15
J., Bhattasali, S., Zhang. S., Franzluebbers, B., Luh, W.-M., Spreng, R. N., Brennan, J. R., Yang. Y., Pallier, C., and Hale, J. (2022). Le petit prince multilingual naturalistic fmri corpus. Scientific data, 9(1):330.
Malik-Moraleda, S., Ayyash, D., Gallée, J., Affourtit, J., Hoffmano, M., Mineroff, Z., Jouravlev, O., and Fedorenko, E. (2022). An investigation across 45 languages and 12 language families reveals universal language network. Nature Neuroscience, 1014)11014
12

---

Manning, L. and Thomas-Antérion, C. (2011). Mare Dax and the discovery of the lateralisation of language in the left cerebral hemisphere. Revue Neurologique, 167(12):868-5) 872.
Matchin, W. and Hickok, G. (2020). The Cortical Organization of Syntax. Cerebral Cortex, 30())1481-1498;
McKinney, W. et al. (2010), Data structures for statistical computing in python. Proceedings of the 9th Python in Science Conference, volume 445, pages 51-56. Austin, TX.
Merity, S., Xiong. C., Bradbury, and Socher, (2016). Pointer sentinel mixture models. arXiv preprint arXiv: 1.Xx::6699008433
Olulade, O, A., Seydell-Greenwer
A., Chambers, C. E., Turkeltaub, E., Dromerick, A. W., Berl, M. M., Gaillard, W. D., and Newport, E. L. (2020). The neural basis of language devel- opment: Changes in lateralization over age. Proceedings of the National Academy of Sciences, 117738)23477-23343
Ozermov-Palchik, O., O'Brien, A. M., Lee, J., Richardson, H., Romeo, R., Lipkin, B., Small, H., Capella, J., Nieto-Castanme Castannn,nnn, A., Saxe, R., Gabrieli, J. D. E., and Fedorenko, E. (2024). Precision IMRI reveals that the language network exhibits adult-like left-hemispheric lateralization by years of age.
Pallier, C., Devauchelle, A.-D., and Dehaene, S. (2011). Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Scienccs, 108(6))2522-2) 2527.
Pasquiou, A., Lakretz, Y., Hale, J., Thirion, B., and Pallier, C. (2022). Neural language models are not born equal to fit brain data, but training helps. arXiv preprint 2rX.v:2207.03380
Pasquiou, A., Lakretz, Y., Thirion, B., and Pallier, C. (2023). Information-Restrica Neural Lan- guage Models Reveal Different Brain Regions' Sensitivity Semantics, Syntax, and Context. Neurobiology of Language, 4(4):611-63)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Patterson, K. and Lambon Ralph, M. (2016). The Hub-and-Spoke Hypothesis of Semantic Memory. In Neurobiology of Language, pages 765-775. Elsevier.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M.. Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011), Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct):2825 2830.
Penfield, W. and Roberts, L. (1959). Speech and Brain Mechanisms. Princeton University Press, Princeton, NJ.
Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 1532-1543.
Pobric, G., Jefferies, E.,, and Lambon Ralph, M. A. (2010). Amodal semantic representations depend on both anterior temporal lobes: Evidence from repetitive transcranial magnetic stimulation. Neuropyycholoia,, 48(5):1336-1333 1342.
Poizner, H.. Bellugi, U., and Klima, E. (1987). What the Hand reveals about the Brain. MIT Press, Cambridge, MA.
Poldrack. R. A. (2011). Inferring mental states from neuroimaging data: From reverse inference to large-scale decoding. Neuron, 72(5):692-697.
Price, A. R., Bonner, M. F., Peelle, J. E., and Grossman, M. (2015). Converging evidence for the neuroanatomic basis of combinatorial semanties in the angular gyrus. Journal of Neuroscience, 35(7):327--3244
13

---

Pylkkänen, L. (2019). The neural basis of combinatory syntax and semantics. Science, 366(6461):62- 66.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAl blog, 188):9.
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko. (2021). The neural architecture language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45).
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy. J., Geiger, F., et al. (2020). Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007.
Seabold, S. and Perktold, J. (2010). statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference.
Seghier, M. L.., Fagan, E., and Price, C. J. (2010). Functional subdivisions in the left angular gyrus where the semantic system meets and diverges from the default network. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, (555))16809-160
Simony, E., Honey, C. J., Chen, J., Lositsky, O., Yeshurun, Y., Wiesel, A., and Hasson, U. (2016).
Dynamic reconfiguration of the default mode network during narrative comprchension. Nature Communications, 7(1):12141. Number: Publisher: Nature Publishing Group.
Snowden, J. S., Harris, J. M., Thompson, J. C., Kobylecki, C., Jones, M., Richardson, A. M., and Neary, (2018). Semantic dementia and the left and right temporal lobes. Cortex, 107:188-203.
Stromswold, K., Caplan, D., Alpert, N., and Rauch, (1996). Localization syntactic comprehen- sion by positron emission tomography. Brain and language, 52(3):452.473.
Szaflarski, J. P., Holland, S. K., Schmithorst, V. J., and Byars, A. W. (2006). fMRI study of language Iateralization children and adults. Human brain mapping. 2723)2202-212.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M.,
Kale, M. S., Love, J., et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.
Toneva, M. and Wehbe, L. (2019). Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint FxXY:23077.0288.
Tzourio-Mazoyerr N., Bertollot,,,,,,i M., Jobard, G.. Mazoyer, B., and Baciu, M. (2017). Multi- factorial modulation of hemispheric specialization and plasticity for language healthy and pathological conditions: A review. Cortex, 86:314-339
Van Der Walt, S., Colbert, S. C., and Varoquaux, G. (2011). The numpy array: structure for efficient numerical computation. Computing in Science & Engineering, 13(2)222.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Vigneau, M., Beaucousin, V., Hervé, P.-Y,, Jobard, G., Petit, L.,, Crivello, F., Mellet, E., Zago, L., Mazoyer, B., and Tzourio-Mazoyer, N. (2011). What is right-hemisphere contribution to phonological, semantc, and sentence processing? Neurolmage, 54(():577-593.
Wada, and Rasmussen, T. (1960). Intracarotid Injection Sodium Amytal for the Lateralization of Cerebral Speech Dominance: Experimental and Clinical Observations. Journal of Neuro- surgery, 17(2):266228 282. Publisher: Journal of Neurosurgery Publishing Group Section: Journal of Neurosurgery.
14

---

Waskom, M. L. (2021). seaborn: statistical data visualization. Journal of Open Source Software 6(60):3021.
Wernicke, C. (1874). Der aphasische Symptomencomplex: eine psychologische Studie auf anatomis- cher Basis. Cohn & Weigert.
Wildgruber, D., Ackermann, H., Kreifelts, B., and Ethofer, T. (2006). Cerebral processing of linguistic and emotional prosody: fMRI studies. Progress in brain research, 156:249-2666 Publisher: Elsevier.
Witelson, S. and Pallie, W. (1973). Left hemisphere specialization for language in the newborn: Neuroanatomical evidence of asymmetry. Brain, 96(3):641-646.
Wolf. T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020). Transformers: State-of-the-arr natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pages 38-45.
Wood, A. G., Harvey, A. S., Wellard, R. M., Abbott, D., Anderson, V., Kean, M., Saling, M., and Jackson, G. D. (2004). Language cortex activation in normal children. Neurology, 63(6))1035 1044.
Xu, J., Kemeny, S., Park, G., Frattali, C., and Braun, A. (2005). Language in context: emergent features of word, sentence, and narrative comprehension. Neurolmage, 25(3):
Zaccarella, and Friederici, A. D. (2016). The neurobiological nature of syntactic hierarchies. Neuroscience & Biobehavioral Reviews.
Zaccarella, E., Schell, M., and Friederici, A. D. (2017). Reviewing the functional basis of the syntactic Merge mechanism for language: coordinate-base activation likelihood estimation meta-analysis. Neuroscience & Biobehavioral Reviews, 80:646-656.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi. Y. (2019). Hellaswag: Can machine really finish your sentence? arXiv preprint WxXiv:/955.07850
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprini 27X.0.2005.01668
15

---

A Supplementary Tables
Table A.1: List of the 28 English large language models included in the study, grouped by family.
model name
Ppraameters
Laayer
Recurons
gpt2
124M
12
768
gpt2-medium
355M
24
1024
gpt2-large
774M
36
1280
gpt2-xl
1.56B
48
1600
opt-125m
125M
12
768
opt-350m
331M
24
1024
opt-1.3b
1.32B
24
2048
opt-2.7b
2.65B
32
2560
opt-6.7b
6,66B
32
4096
opt-13b
12.9B
40
5120
Llama-2-7b-hf
6.74B
32
4096
Llama-2-13b-hf
13.02B
40
5120
Qwen1.5-0.5B
464M
24
1024
Qwen1.5-1..B
1.84B
24
2048
3.95B
40
2560
Qwenl .5-7B
7.72B
32
4096
Qwen1.5-144
14.17B
40
5120
gemma-2b
2.51B
18
2048
gemma-7b
8.54B
28
3072
stablclm-2-1_66 6b
1.64B
24
2048
stablelm-3b-4clt
2.80B
32
2560
stablelm-2-12b
12.14B
40
5120
Mistral-7B-v0.1
7.24B
32
4096
mamba-1300 130m-hh
129M
24
768
mamba-370m-hn
372M
48
1024
mamba-790m-ht
793M
48
1536
mamba-1.4be
1.37B
48
2048
mamba-2.8b-hf
2.77B
64
2560
16

---

Table A.2: Per ROI tests of the interaction between hemisphere and model size. The first column the name of each ROI. The second column gives the p-value the two sample t-test between the set of slopes of all the voxels of the left vs. the right corresponding ROI, whose means and 95% confidence intervals are depicted in Fig. 7b. The third column presents the correlation between the difference in performance of the models (the r-scores) hemisphere as a function of the logarithm of the number of parameters, as described in Fig. 7c. The last column gives the p-value of the correlation. These two measures of interaction lead to the same conclusions: the interaction is not significant for BA44 and TP (marginal in the latter case), and highly significant for all the other ROIs, namely aSTS, PSTS, AG_TP] BA45 and BA47.
the left hemisphere minus the right
ROI
p-value of two
correlation L-R
p-value
sample t-test
log( (pprrmmttes)
TP
4.1e-02
0.36
6.2e-02
aSTS
7.3e-11
0.85
1.0e-08
PSTS
3.5e-09
0,82
1.2e-07
AG TPJ
9.3e-22
0.84
1.7e-08
BA44
4.8e-01
0.15
4.6e-01
BA45
5.7e-07
0.73
9.4e-06
BA47
1.4e-05
0.56
1.9e-03
17

---

B Supplementary Figures
Ocdstial Picc
Heseills Gynu (irlufer Ri 421
fargerate
Pedeta Opercile Canas
Open-iie
Tmmorn fosterm Cartex, possetor draig
y=443
x=0
z
Rastippaaame pal Gyru, posie drrsgs
(ingual
Paraniipramp frrrtal Oibiral Cotas
Prosions Carta
Cinulate pocarterdiae
drigilati Peachhuuee cynis
jutapodh Loque Canes
Trental
Latea Derioial cont: interor drads
Left Hemisphere
0.6
Angiar Gynus
pastelor draip
Sasnmargin Superit Poticta
Infonio Tumpora Satus ponppooodlillllo part
Intetir
Ho Temronal denic ponponoddeit
draips
0.4
Nilale lonsuta Conu- anarto deciss
superor Cynu, ensetor
Romporal Pile
yper or herral Syrus pin sperca ars
Syn-
Supoitor hrntal
Frental Pile
0.2
0.4
0.6
Right Hemisphere
0.00 0.05
0.10
Figure B.1: Inter-subjects correlations (model free) in homologous cortical regions of the left and right hemispheres. (a) The 48 cortical areas from nilearn' cort mettmmaxooobthte ttrrtthrr-- -mm version of the Harvard-Oxford atlas. (b) Relationship between right and left inter-subjects correlations (dots correspond to regions from the atlas) (c) Differences in correlations between left and right regions. The number of stars next to each indicates an estimate of the level of statistical significance,
assessed with two-sample t-test (+:p < 0.05, ** < 0.01, *** < 0.001).
18

---

120
urdem vestea 10283
norma deraity
1r emorcdingg 301c
1r embecdings 1024d
427
large language models
626
925
327 number paramaters
brain correlation
Liama
Gamma
StableL M
Mistral
Namba
0.28
0.26
0.24
10"º10"
10
10
10
10
1001
10'"10"
number parameters
Figure B.2: Performance of various models in predicting fMRI brain time-courses. Same as Fig. but for the whole brain volume.
0.46
GFF2
b.
GPT2
0.75
0.64
0.20
Canme
Quen2.5
0.255
Mara
Mooa
Naro3
0.29
0250
0.240
30
12a number of parameters
1644
parameters
Figure B.3: Scaling law of IMRI encoding models, by family. (a) Average over the 25% most reliable voxels (b) Average over the whole brain volume.
19

---

a.
Qwen1.5
StabieLM
brain corclation
0.40
0.25
layer depth
@wenl15
Gemma
StabkIEM
0.275
0,250
brain comelatton
0,200
0.175
layer depth
Figure B.4: Brain correlation function of the relative layer depth, split by family. The relative layer depth corresponds to the depth of a given layer relative to the input and output layers: the first layer is the non-contextual embedding layer, position O, and the last hidden layer is at position 1. Considering (a) the 25% most reliable voxels or (b) the whole brain volume.
b.
3.4
0.275
cometationn
0.270
0.23
0.53
0245
25
15 perslexity
10
hallzswog
-11
brain correlation
0.2775
0.2725
0.2700
4
60
1034
70
75
85
parameters
perplexity
hellaswag
Figure B.5: Brain correlation and performance natural language tasks. Same Fig. 5, but for the whole brain volume.
20

---

a.
0.007
0.82
0.006
brain carmistion
brain cordaation
0.20
0.25
0.001
0.000
0.24
10f
10º
1023
10a
10
104
number of parameters
number
d.
0.48
0.025
0=::1.0 -2
brain conrlatiin
0.020
0.46
0.015
0.005
0,42
0.000
10a
10'
10
parameters
number of parameters
Figure B.6: Emergence of left lateralization with the size of the encoding models. Same Fig. but, top panels (a) and (b), for the whole brain volume, and bottom panels (c) and (d) considering the best 10% voxels of the left hemisphere vs. the best 10% voxels of the right hemisphere.
a.
GPF-2
DPT
Unma 2
Owen1.5
Demma
LMM
Mistral
Mamba
brain correlation L. R
6.01
-0.02
-0.03
relative layer depth
GPT-2
OPT
Liama 2
Owen1.5
Gemma
StableLM
Mistral
Mamba
brain corrolation
0.000
-0.005
-0.010
relative layer depth
Figure B.7: Left-right difference in brain correlation as a function of the relative layer depth, split by family. Considering (a) the 25% most reliable voxels or (b) the whole brain volume.
21

---

b.
0.78
0.010
p-1.6-20 00
0.76
0.005
brain conrelation
normalaeed
0.74
nommllzzd
0.72
-0.100
0.70
-2015
-0020
104
10
1654
number
number parameters
d.
brain score function model size
0.50
p-1.70 41
Sope of the R difference, in
0.008
Pi 5.6e 01
averaged accross models
0.006
brain correlation
0.40
0.35
0.004
0.002
0.25
0.20
2010
-6.02 1. difference inter-subject comreaation
inten-subject correlation
Figure B.8: Influence of inter-subject correlations on brain scores and growth in asymmetry. (a) and (b): Same as Fig. 6, but brain correlations are normalized by inter-subject correlations. (c) Brain correlation (average over all models) as function of the inter-subjects correlation in each of the 48 parcels of the Harvard-Oxford atlas depicted in Fig. B.I. (d) Slope the relationship between the L-R difference in r-score and the logarithm of the number of parameters, function of inter-subjects correlation. Each square corresponds to a parcel the Harvard-Oxfol atlas.
(ns)
(**)
(ns)
11*
0.65
brain corriatioon
0.55
0.50
0.40
0.30
100
10P
102a
10P
20
102
102
1031
10f
10P
1014
10f
10P
1638
109
10P
10th
10P
10
number parameters
Figure B.9: Analysis by Regions of Interest. Brain correlation as a function of the number of parameters (in log scale), for all 28 models, for each ROI, for each hemisphere. The horizontal dashed lines represent the mean inter-subjects correlation in each corresponding ROI (along with 95% confidence interval).
22

---

GPT
Q.en1.5
0.01
r=
0096
trained untrained
8.30
-03
p 1.00 02
0.00
0.010
c
-0.01
0.005
0.02
0,000
-0,005
-0.04
-0.010
p 9.8e
p= 6.5e
10a
10a
10ta
number parameters
brain correlation
0.40
0.35
0.20
brain correlation: R-R
0.01
0.00
-0.01
100%
percent of total training (1 epoch)
Figure 1.10: Impact of model training. (a) Left-right difference in brain correlation as a function of the number of parameters (in log scale), for two model families (GPT-2 and Qwen1.5), before (gray dots) and after (blue dots) training. (b) Brain correlation (top) and left-right difference in brain correlation (bottom) untrained model (0%) to the fully trained model (100% corresponds to one epoch through the 300B tokens training set). All results were computed the 25% most reliable voxels.
function of the amount of training, for the Pythia-6.96 model. From the
23

---

b.
0.88
0.44
75%
34 RA
C.
Dec Pole
Hrach's 's Gyrua (ndluds HS PQI
Panisal Opentu ar Cortes
Funtal Ope Coroor
Temeera Fulftem Cates, acater duison
lcm
y=
x=O
brattppocamall Gynus,
Liigus carus
Parstippzammell Gynis, dive
Pornil gratia
Precunscus Cortex
d.
Cingu Gytss,
staccnulate
Jasapos Bonal Lobu Corise
licaralaati Cortor
Latenal Ocssttll Corne, Latal Ocs
Left Hemisphere
0.6
Angular Gyrus
Sup Sxpaamagitn Oyrss, atterne 0lcsion
Inferie lemoaa Cyras temiefoscco Dyrus inflnit lor paral oyrss ante or dira on Natel
0.4
Moddo lonppor a Oy ter dhis susorict Tomper al by fos toe dirs
Ineeiar frantal Dynu, encertrattynis
fecd Promeal Gyrus
thau
Prrita itol
0.2
0.4
0.6
Right Hemisphere
f.
h.
model name
7laaers
Vineurnss
0.450
8xiplab/gnt2-tiny-chieee
12M
312
comilation
popplab/gptt-bas-chheee
102M
12
312768
Qwenl.5-0.5B
464M
24
1024
Qwenl.5-1 5-8.BB
1.84B
24
2048
0.350
Qwcml_5-4B .5-4B
3,95B
40
2560
Qwenl.5-78 15-BB
7.72B
32
4096
Qwenl 5-14B
14.17B
40
5120
Qwen2-0.5B
494M
24
896
0275
10'
10'
10"
10t
Qwen2-1.5B
1.54B
28
1536
number of parameters
Qwen2-7B
7.62B
28
3584
2002
78ncOm vectire 10245
7211000 mnlaedbros 352
0.025
romalize
0.9
0.1
0.3
0.4
0.5
3.6
brain comreattoo
0.800
10
102
100
2014
number of parameters
Figure B.11: Results from Chinese participants and large language models trained on Chinese data. Analyses performed on all 35 Chinese participants from the multilingual fMRI dataset Le Petit Prince. The average subject has 26050 voxels. (a.b) Same as Fig. (a.b). (c,d.e) Same as Fig. B.1 (a,b,).. (f) List of all large language models used in this Chinese study (pretrained with an autoregressive objective on Chinese language data), as available on the Hugging Face hub. (g.h) Same as Fig. (a, b). (i) Same as Fig. 6 (b).
24

---

a.
0.79
100%
0%
e.
Supracainatien Corter
temporas
Cyrus
Panum Feans
Opencier Eorte
Fonbal
Rmpponl optpinl Redform Contex
y3.43
x=O
Tomeura furm Cattox, sttriir dxtain
8
Berahiposcam Gytas, atteter duition
Cureal Cortes
Cingulate Gynu, costeree
distion
Subca los Cortes
Mecapostoonl Cortes
Weter oos piul corte,, superor dxition
Left Hemisphere
Supmmaggie angiaal Gyrs, actteler dxition
0.4
Poocertrs Syras
inter Ireparal Gyn-s pettetor duliion
Inluner
Mesie Tempoial al dyn.s, cesteter dititon
Supet lumpo of Genis acstatoo dxstion
pars
Rrcerbra Cytus
0.2
infotoo Forita Cyus, (ors anjuias
Incuter Coftes
Right Hemisphere
0.00
0.05
0.10
f.
0.40
model name
Npammmeters
Players
Placurous
0:38
ClassCa//gi22-aas--fen Qwen2-0.5B
124M
768
boin candati
0:36
494M
24
896
Qwen2-1.55 1.5B
1.54B
28
1536
0.32
Qwen2-7B 7B
7.62B
28
3584
9:30
326
10t
number of parameters
TATOUET VeSDo 2002
TeTOUR Vester
0.010
webedSing: 1034d
corretetion
0.005
normallzed
0.2 brain correlation
10
rumber of parameters
Figure B. 12: Results from French participants and large language models trained on French data. Analyses performed on all 28 French participants from the multilingual fMRI dataset Le Petit Prince. The average subject has 27773 voxels. (a.b) Same as Fig. (a.b). (c,d,e) Same as Fig. B.1 (a,b,c). (f) List of all large language models used in this French study (pretrained with an autoregressive objective on French language data). available on the Hugging Face hub. (g.h) Same as Fig. 2 (a. b). (i) Same as Fig. 6 (b).
25

---

a.
aub CN057
9AD ENCO0
BA0 EV052
1L0
SNDE2
breins Mheincorwwaan cormitiio
0.0902
0.0550
9.2000
0.0022
0.0002
0.0325
01223
0.0200
3.3950
0313
9.252
6.0723
3,3400
0,0700
0112
204
127
2644
30
10a
10
10
12a
20
12
101
10
30a
10'
1214
rumber of parometers
b.
0.140
0.2300
0.045
0.235
U14th COTTMAISE
0.135
0.1050
0.230
0.1.0
0.065
0.225
3.2975
0.075
3.3925
0.072
3,3000
0.100
0.052
104
164
30F
30
10a
10
10
1214
10a
13R
number of parameters
C.
2M127
MAD: EN550
sD 3N053
E1.02
0.008
5,0023
0.024
2.023
0.001
2.034
Brahn crin-come
0.008
5.0012
-0.0026
2.0003
0.005
2002
0.003
5.0005
-0
5,0025
2.02
0.002
-2.0013
-0.0022
10
122
12a
10'
10
10
121
1214
12
700
2644
20
12a
1214
number of parameters
d.
0,300
0.265
brain comm00tton
3.18
0.20
0:375
0.200
1.20
0.355
2.55
5.185
0.200
0.200
0.33
2.14
300
105
164
10
100
30
1.2
107
30
104
13P
10
101
131
number of parameters
e.
0. 0.0325
Rab
0.013
F1
1áb
5.006
sub
Lub C102
0.0125
TMD DND62
0.00000
2.035
0.0130
0
0.0075
0.0250
0.0020
0.0125
0.0025
0.0100
0.0020
0.0275
0.002
0,025
0.0075
10
121
1314
13a
10'
1014
16"
125
10
125
200
26'
20P
125
30
rumber paromaters
Figure B.13: Individual analyses performed on five English participants. Each column cor- responds to one participant, sub-ENO57, sub-EN058, sub-ENO59, sub-ENOG and sub-EN062 respectively. Nine large language models were used: all the models from the GPT-2 and Qwen1. 1..5 families used in the main study (see Table A. ranging from 124M to 14.2B parameters. (a) For each individual, brain correlation on the whole brain volume as a function of the number of parameters (in log scale). (b) Same (a), but for the left hemisphere (red) and for the right hemisphere (green). (c) Difference between left and right hemispheric brain correlations. (d) and (e) Same as (b) and (c) but considering the best 10% voxels of the left hemisphere vs. the best 10% voxels of the right hemisphere.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The novel contribution of this work, clearly stated in both the abstract and at the end of the introduction, is the discovery of left-right brain asymmetry in the predictive power of NLP models when their complexity increases. This is clearly supported by the experimental results presented and discussed in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
There a dedicated subsection in the Discussion & Conclusion section, right before the References that discusses two limitations of our work (use of group analysis rather than individual analyses, and limitation to a single language:English)
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The Methods section is devoted exactly that. Moreover, we use dataset publicly available, and all the code written for the project is shared with the reviewers and will be released publicly.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The fMRI data comes from a publicly available dataset, and we indicate the exact version and URL of the dataset used in our study. Code will be released publicly GitHub. order to preserve anonymity, we share the code as zip file with the submission.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. wttpssscccppcccccccttecc.cccccct
Mees/CCooSubbii issiiss ssisssiin ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification: of Methods section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Throughout the paper, whenever relevant, figures include error bars well as statistical information such as correlation coefficient and p-value. We provide confidence intervals, well the descriptions of how these were computed, notably the factors of variability that these error bars capture. Explanations can be found in the captions of the figure, in the main text in appendix (cf Table A.2).
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: ef section 2.3 Computer code in the Methods section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips. httpppss/scppppttttttttttiitiaie ines?
Answer: [Yes]
Justification: We reviewed the NeurIPS Code of Ethics and found that our work respects all the points.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
30

---

Justification: As we mention in the abstract, our finding reconciles results from neuroscience and machine learning, which positive impact for the scientific community. Yet, we cannot think of any industrial application, nor any direct risk or negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available fMRI corpus, Le Petit Prince. We indicate the exact version that we use, as well as the URL where to find it, on openneuro.org orgg, and cite the paper accompanying the release of this dataset. As for the code and pretrained models, we clearly list all the pretrained models and packages that we use, their versions, and we eite the relevant corresponding work for each of them. See in particular the Methods section.
31

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the code is shared along with full explanations of the procedure to exactly reproduce the results of the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
32

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "okjVTY9CaJ", "reviewer": "Reviewer_Wj1c", "summary": "The paper studies whether encoding fits with LLMs onto fMRI data can be used to find left lateralization, the idea that language is lateralized to the left hemisphere. This is a well-known property of language localization in many humans. They also show that increasing model size improves fit from their encoding models.", "questions": "* Why did you also include Mamba? I’m not sure it adds anything or detracts anything but I’m just curious about the choice.", "limitations": "I believe all reasonable limitations were addressed with regards to this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "* A new fMRI dataset that can serve as additional analysis for LLMs. \n* Incorporation of a large family of LLMs never studied before in previous work including LLaMA-2, Mamba, etc.\n* Likely the first (to my knowledge) to directly test the correspondence between LLMs and lateralization.", "weaknesses": "* Although the motivation of this paper is reasonable, I think the presentation can be significantly improved. \n    * The paper should make an argument about why lateralization is an important property of brain fits with LLMs. I think this was missing.\n    * A small nitpick: From my understanding, I can’t see any data from right hemisphere regions in Schrimpf et. al. (2021) [1]. The paper is very careful in acknowledging left-lateralization (see Page 2 of the Schrimpf paper). \n    * The introduction should present some more details on the setting. What language models are used? What data? This is covered later but is missing from an introduction that should introduce more of the paper. Also the introduction doesn’t even mention the other contribution of the paper on model size correlating with neural fit. \n* The novelty and takeaways of this paper are unclear to me.\n    * First, the paper claims that larger language models have better fits. Schrimpf et. al. (2021) [1] already establishes that. This is acknowledged in the conclusion/discussion section. The authors also refer to Antonello et. al. 2024 which seems to establish this as well. How would the authors characterize the difference in the takeaway of their paper vs the prior takeaway? I’m not particularly satisfied with the argument that much larger models were used or that a logarithmic scaling law was identified. \n    * Caucheteux and King [2] discuss lateralization at length in their paper. Refer to page 3 where they show that language embeddings have significantly better left-lateralization over right-lateralization (R±0.01, p < 10^-14). This may not use the controls presented in this paper but still points to the same result. \n    * I think both of these need to be discussed at more length. The contribution of this paper should be better characterized. I don’t see the novelty of this work even in regards to lateralization.\n* Baselines\n    * The paper doesn’t consider the baseline comparison with randomly initialized models. Why? I think this is a very important baseline for characterizing architectural bias and this is done in previous work.\n\n[1] Schrimpf et. al. The neural architecture of language: Intergrative modeling converges on predictive processing. PNAS 2021. \n[2] Caucheteux et. al. Brains and algorithms partially converge in natural language processing. Nature communications, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Here is a point-by-point response to your comments.\n\n> The paper should make an argument about why lateralization is an important property of brain fits with LLMs. I think this was missing.\n\nStudies correlating word embeddings or LLMs activations and fMRI data have produced strikingly bilateral results. One potential explanation is that brain scores are essentially driven by semantic representations which are supposedly represented in a very distributed fashion across both hemispheres. Our work shows that the symmetrical results are due to the use of models with small numbers of parameters and that when the number of parameters increases, the left-right asymmetry becomes larger and larger. This non obvious empirical observation opens several questions for further investigations, notably what aspects of large LLM relative to small ones make them more asymmetrical.\n\n> A small nitpick: From my understanding, I can’t see any data from right hemisphere regions in Schrimpf et. al. (2021) [1]. The paper is very careful in acknowledging left-lateralization (see Page 2 of the Schrimpf paper).\n\nSchrimpf et al (2021) show results from both hemispheres in Fig. S3 of their supplementary materials. Responses on both sides are remarkably similar and the caption states that \"The distribution of predictivity values across the language-responsive voxels [...] are similar across regions, and between the LH and RH components of the network\". The authors do not report any quantitative comparison between hemispheres. While they acknowledge the well-known fact that language is left lateralized, surprisingly, they did not discuss the lack of lateralization in their results. Our work replicates their findings with small models but shows that it breaks down with larger models. \n\n> The introduction should present some more details on the setting. What language models are used? What data? This is covered later but is missing from an introduction that should introduce more of the paper.\n\nThe introduction only contained sparse information about the exact list of language models that are used, but all such information is provided in great details both in the Methods section (that follows the Introduction) and the Appendices. As the final camera-ready version of the paper allows for one extra page, we will be able to present more details as suggested.\n\n> First, the paper claims that larger language models have better fits. Schrimpf et. al. (2021) [1] already establishes that. This is acknowledged in the conclusion/discussion section. The authors also refer to Antonello et. al. 2024 which seems to establish this as well. How would the authors characterize the difference in the takeaway of their paper vs the prior takeaway?\n\nOur discussion simply mentions that we replicate these previous findings and devotes more space on the growing left-right asymmetry which is a genuine novelty. \n\n> Caucheteux and King [2] discuss lateralization at length in their paper. Refer to page 3 where they show that language embeddings have significantly better left-lateralization over right-lateralization (R±0.01, p < 10^-14). This may not use the controls presented in this paper but still points to the same result.\n\nThank you for pointing to the left-right test by Caucheteux & King (2022), which we had missed. Note that although they test it, they do not really discuss nor comment on it in their paper.\n\nNevertheless, we believe that our paper goes well beyond showing a left-right difference in a single model. Given one model, if the data has higher signal-to-noise ratio in the LH, one expects an encoding model to better fit the LH compared to the RH. This is in fact the case in the Caucheteux & King (2022; Fig. 2d).\n\nIt is a legitimate concern that if the signal-to-noise is higher in the left hemisphere, then the brain correlations, as predicted by the encoding models, should also be higher in general. But, given that the data is constant across models, it is unclear why this would affect the slope in comparing brain correlations and number of parameters. We did some extra analyses (see pdf)  showing that, although the mean brain correlation depends on signal-to-noise ratio, the slope cannot be explained by this factor. \n\n> The paper doesn’t consider the baseline comparison with randomly initialized models. Why? I think this is a very important baseline for characterizing architectural bias and this is done in previous work\n\nFig. 2a of the submission presents the distributions of r-scores from various models, including distributions from random (fixed) embedding models. Pasquiou et al (2022, ICML) have shown that this type of model yields brain correlations which are as strong or stronger than untrained language models. Nevertheless, following the reviewer’s comment, we computed the brain scores for untrained versions of the four variants of GPT-2. We replicate Pasquiou et al.’s observation, that is, all four untrained models performed below the 1024d random baseline which achieved an average brain score of 0.182 (on the 25% most reliable voxels). For example, gpt2-medium, which has the same number of dimensions, yields a brain correlation of 0.168 (vs. 0.417 for the corresponding trained model). While the left-right score difference increases with the number of parameters (r=0.99, p=8.3e-3), the relationship breaks down and is flat in the case of the untrained models (r=-0.03, p=0.97).\n\n> Why did you also include Mamba? I’m not sure it adds anything or detracts anything but I’m just curious about the choice.\n\nMainly out of curiosity. It is the first time that a large language model not based on the Transformer architecture has competitive performance on par with more traditional Transformer-based language models. As discussed in the Results section, the Mamba family has similar performance as encoding models of brain functional data, which was not a priori obvious.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HGB8lsLHQU", "reviewer": "Reviewer_jQCd", "summary": "Recent studies on language processing in the human brain using fMRI data and language model embeddings have shown that both hemispheres are involved in language processing although many previous studies have indicated a left lateralization. The authors aim to reconcile these findings. They use embeddings from language models of different sizes to predict naturalistic fMRI data and show that as the number of parameters or performance of the models increases, the respective embeddings are more predictive of activity in the left hemisphere vs. in the right one. They also show that this pattern holds in various regions of interest.", "questions": "I have mentioned my questions and suggestions in the Weaknesses section above.", "limitations": "The authors have identified the limitations of their analyses and I do not foresee any negative societal impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The main novel contribution of this work is showing the clear dependence between model capacity or performance and predicted left lateralization of language processing. Many previous studies, including some that have been cited, demonstrate left lateralization to a certain extent (ex: Fig 3 from Cauchetaux and King (2022)) and have shown that predictivity scales with language model capacity (ex: Fig 2 from Schrimpf et al. (2021)). However, I appreciate the thoroughness and clarity of the authors’ analyses and believe that it provides additional evidence to back a key finding in neuroscience.\n\n- Quality: Experiments performed to support claims are very thorough and well-motivated.\n\n- Clarity: The paper is well-written and easy to understand. One should be able to reproduce the results presented using the paper and associated code.\n\n- Significance: As mentioned above, although previous studies have shown left lateralization to a certain extent and that predictivity scales with language model capacity, I believe that this work is still important since it clearly shows that more left-lateralization emerges as the models are scaled up.", "weaknesses": "This is a strong submission in general. I have the following suggestions/questions to improve it:\n\n1. The second paragraph of the introduction (and the abstract) mentions that many recent works have discovered strikingly symmetric brain maps for language processing. To the best of my knowledge, many of these studies conclude that both hemispheres are involved in language processing without assessing left-lateralization since other studies had contended the role of the right hemisphere altogether. If one looks at their figures (ex: Fig 2 from Toneva and Wehbe (2019) or Fig 3 from Cauchetaux and King (2022)), the left hemisphere is generally predicted better than the right hemisphere. Therefore, I think that this paragraph can be modified to better reflect these results.\n\n2. I understand the need for reducing the computational burden of the study and the nuances mentioned in the limitations section. However, for a subset of the participants, it would be great to see individual participant-level results to make an even more convincing argument. This is mostly to rule out any artifacts associated with using an average subject and not to make inferences about inter-individual variability.\n\n3. For encoding any given word, are the embeddings computed by only inputting the previous words or using the full sentence/text? In other words, are the embeddings capturing future information due to the full sentence/text being encoded at once? If they are capturing future information, that is at odds with how the data was collected since words were sequentially presented to participants.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Here is a point-by-point response to your comments.\n\n> The second paragraph of the introduction (and the abstract) mentions that many recent works have discovered strikingly symmetric brain maps for language processing. To the best of my knowledge, many of these studies conclude that both hemispheres are involved in language processing without assessing left-lateralization since other studies had contended the role of the right hemisphere altogether. If one looks at their figures (ex: Fig 2 from Toneva and Wehbe (2019) or Fig 3 from Cauchetaux and King (2022)), the left hemisphere is generally predicted better than the right hemisphere. Therefore, I think that this paragraph can be modified to better reflect these results.\n\nThanks for pointing that out. We apologize for missing these passages and will modify our introduction accordingly.\n\n> I understand the need for reducing the computational burden of the study and the nuances mentioned in the limitations section. However, for a subset of the participants, it would be great to see individual participant-level results to make an even more convincing argument. This is mostly to rule out any artifacts associated with using an average subject and not to make inferences about inter-individual variability.\n\nWe agree this is an important point, but we will not be able to conduct this work in the time-frame. We have modified our paragraph in the Limitations section to better acknowledge the interest of performing the same analyses at the individual level, which we will do in the future.\n\n> For encoding any given word, are the embeddings computed by only inputting the previous words or using the full sentence/text? In other words, are the embeddings capturing future information due to the full sentence/text being encoded at once? If they are capturing future information, that is at odds with how the data was collected since words were sequentially presented to participants.\n\nIndeed, the embeddings are computed autoregressively, taking into account only the past of a given word, and not future information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4r3N3mLXUX", "reviewer": "Reviewer_d74H", "summary": "The authors investigate whether larger-parameter language models better predict left versus right hemisphere brain responses (recorded during listening of a naturalistic story, via fMRI), motivated by left-lateralized processing of language in most individuals. They indeed find that larger models better predict left hemisphere brain responses, compared to right hemisphere responses. In addition, the authors have two separate findings that mainly replicate prior work: larger models better predict brain responses, and the prediction performance correlates with the language model's ability to perform natural language tasks.", "questions": "- Pertaining to the issue of group-level averaging, is it true that the left-hemisphere generally has higher reliability? Was that controlled for in any way in the analyses (as far as I understand, the authors analyze the top 25% most reliable voxels, yielding approximately a similar number of voxels in both hemispheres (~3K). What are the average reliability values per hemisphere? If the authors subset left and right hemisphere voxels such that they are matched on reliability, are the primary model-brain lateralization findings still valid?\n- Are the performance vs. brain correlations (Figure 5) equally strong in both hemispheres? Do the left hemisphere brain activations contain more information about task, relative to the right ones?\n- Can the authors clarify what they mean by \"The right hemisphere is usually “hidden” because it is inhibited by the left hemisphere in healthy people\"? (line 285)?", "limitations": "The authors do discuss limitations briefly, but I am not sure I agree with the conclusions that follow. For instance, in lines 299-300 the authors mention the group-level approach, but conclude that assessing inter-individual variability would require \"However, this would require a random sample of the population, not only right-handed participants\". Why is that true? Inter-individual variability can still be investigated in right-handed individuals? It is true that it would be useful to assess the hemispheric dominance of individuals -- that could be done by e.g., assessing the consistency of brain responses within an individual, across runs (other approaches exist as well).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The primary finding of the paper, that larger language models better predict left hemisphere brain responses, is novel. \n- The paper is well-written and well-cited. The structure of the paper is intuitive, making it easy to follow. \n- The paper has a good number of control experiments.\n- The paper provides nice replication of former work in this domain.", "weaknesses": "- The paper averages across different individual's brain responses (n=48) in the template brain space. Individual activations to language differ from participant to participant, and there is no direct voxel-to-voxel correspondence (see e.g., https://pubmed.ncbi.nlm.nih.gov/32160565/). Hence, it is somewhat unclear whether the primary finding of brain-model lateralization can be explained away by methodological issues. Imagine that individuals generally have a higher voxel-to-voxel correspondence in the left hemisphere compared to the right -- would it be true that large models, given higher expressivity, would be able to better predict those left-hemisphere voxels? (see Questions for more specific questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Here is a point-by-point response to your comments.\n\n> Pertaining to the issue of group-level averaging, is it true that the left-hemisphere generally has higher reliability? Was that controlled for in any way in the analyses (as far as I understand, the authors analyze the top 25% most reliable voxels, yielding approximately a similar number of voxels in both hemispheres (~3K). What are the average reliability values per hemisphere? If the authors subset left and right hemisphere voxels such that they are matched on reliability, are the primary model-brain lateralization findings still valid?\n\nThanks for bringing this up. The description of the data itself deserves more attention, and the question of whether the effect is simply due to a difference in the left-right signal-to-noise ratio is an important one. We will discuss these two aspects in more detail in the camera-ready version, along with the addition of two new figures shared in the rebuttal pdf that show new analyses.\n\nFirst, with respect to the question of group averaging, we show that the left hemisphere does indeed have a higher reliability in general (see Fig. 1 of accompanying pdf), although the inter-subjects reliability follows the diagonal in left vs. right plot as a general trend. Now, importantly for the discussion, this could explain why for a single encoding model there is a better correlation in the left than in the right hemisphere, but this could not explain the left-right difference scaling law as a function of the number of parameters, since the quality of the data is a constant across models. In short, Fig. 2 shows that (i) the mean brain correlation follows the reliability of the signal in the different regions of interest, but that (ii) the slopes do not generally follow such a trend. This point is also discussed at length in another reviewer thread. \n\n> Are the performance vs. brain correlations (Figure 5) equally strong in both hemispheres? Do the left hemisphere brain activations contain more information about task, relative to the right ones?\n\nThe relationships between performance (hellasawag or perplexity) and r-scores, split by hemisphere, show the same increasing left-right difference as the version with (log of) number of parameters on the x-axis, that is, the better the performance, the more marked the asymmetry. These graphics will be added as supplementary figures of the paper.\n\nMore generally, we think that it is important to further investigate the relationship between the performance on a given task (to name but a few: syntactic comprehension, sentiment analysis, social understanding) and the difference in brain score in a given hemisphere or region. We are working on this, but this goes well beyond the present paper.\n\n> Can the authors clarify what they mean by \"The right hemisphere is usually “hidden” because it is inhibited by the left hemisphere in healthy people\"? (line 285)?\n\nThis affirmation refers to the concept of interhemispheric inhibition according to which one hemisphere can prevent concurrent processing by the opposite hemisphere. It is one of the theories proposed to explain the capacity of the right hemisphere to take over linguistic functions in aphasic patients with left lesions (for example, Tzourio-Mazoyer et al., 2017, cited in our paper, wrote:  “In typical brains, inter-hemispheric inhibition, exerted from the LH to the RH, permits the LH to maintain language dominance. In pathological conditions, inter and intra-hemispheric inhibition is decreased, inducing modifications on the degree of Hemispheric Specialisation and of language networks.”). We will tone down this assertion to make it clearer that it is an hypothesis.\n\n> The authors do discuss limitations briefly, but I am not sure I agree with the conclusions that follow. For instance, in lines 299-300 the authors mention the group-level approach, but conclude that assessing inter-individual variability would require \"However, this would require a random sample of the population, not only right-handed participants\". Why is that true? Inter-individual variability can still be investigated in right-handed individuals? It is true that it would be useful to assess the hemispheric dominance of individuals -- that could be done by e.g., assessing the consistency of brain responses within an individual, across runs (other approaches exist as well).\n\nIt is true that interindividual variability could be investigated only in right-handed participants. What we had in mind is that if one wanted to assess if the LLMs correlate more strongly with the dominant hemisphere of humans in general, it  would make sense to generalize to the whole population, rather than a sub-population. Anyway, because this issue is not central to the paper, we will remove the sentence. A more relevant focus for this section is “what could we learn from individual analyses?”, and we will expand on that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fDc8Ed9MjN", "reviewer": "Reviewer_LDde", "summary": "The paper studies how well 28 large language models (LLMs) predict fMRI activity of human subjects listening to an audiobook. First, they observe a scaling law; the neural predictivity of LLMs increases linearly with the logarithm of the number of model parameters, a result consistent with prior work. Second, they show larger models predict left-hemisphere activity better than right-hemisphere activity, and this left-right asymmetry increases with model size.", "questions": "Just a minor suggestion to fix typos:\n1. Hyperlinks to appendix figures do not work correctly. E.g., \"Fig. B.1\" (Line 162), also: Lines 173, etc. These are the hyperlinks that start with \"Fig B.\"", "limitations": "Limitations not mentioned in the paper:\n1. See Weaknesses 3 and 4.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written clearly and has put effort into sufficiently explaining the methodological details.", "weaknesses": "From most to least significant:\n1. For the paper's main contribution, i.e., asymmetry in Left-Right neural predictivity that increases with model size (Figure 6), the paper does not provide sufficient experimental/theoretical analyses into possible reasons, nor an adequate attempt into possible explanations/interpretations.\n2. The growing L-R asymmetry result may suggest qualitatively interesting differences between larger vs smaller LLMs, as briefly mentioned in Lines 256-260 in the paper. However, an alternative reason could be that right-hemisphere activity is just lower in general (as shown in many prior studies), or more noisy / less consistent across subjects (especially since this paper uses a group-average of fMRI activity).\n- The L-R difference does not seem qualitatively interesting, because both trends in Fig. 6a are straight lines. This seems to simply hint at left-hemisphere activity being easier to predict. If the left-hemisphere line were exponentially increasing instead, that may hint at a qualitatively interesting difference between larger vs smaller models. But instead, we see two straight lines. In fact, the L-R difference seems to already be present with models with 350M parameters, and the gap is only linearly widening as model size increases.\n3. For the growing L-R asymmetry result, there is no significant effect in many key regions of the language network (IFG, temporal pole) (Figures 4 and 7) (Lines 225-227). The strongest effects seem to occur in AG, which some consider not part of the language network [1]. This is a relevant weakness since this paper focuses on the language network.\n4. They compute a group average of fMRI activity across subjects before using LLMs to predict fMRI activity (Line 64), and identified regions of interest (ROIs) in the language network using anatomical locations (Line 95-98), rather than functional localization. However, these methodological choices may have issues when studying the language network (see Box 1 of [1]). Because of inter-individual variability in the precise anatomical locations and sizes/shapes of functional areas in the language network, any voxel defined in a common anatomical space often corresponds to different functional areas across individuals. Hence, using group averages and anatomical ROI localization can lead to the blurring of neighboring areas and information loss [1].\n\n[1] E. Fedorenko, “The language network as a natural kind within the broader landscape of the human brain,” Nature Reviews Neuroscience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Here is a point-by-point response to your comments.\n\n> The growing L-R asymmetry result may suggest qualitatively interesting differences between larger vs smaller LLMs, as briefly mentioned in Lines 256-260 in the paper. However, an alternative reason could be that right-hemisphere activity is just lower in general (as shown in many prior studies), or more noisy / less consistent across subjects (especially since this paper uses a group-average of fMRI activity).\n\nThanks for raising the point of left vs right signal-to-noise in the data. Indeed, one intuitively expects higher brain correlations in regions where the signal-to-noise ratio is better and therefore predicts left-right asymmetries based on this point. First of all, note that here, we compare a large variety of models using the same data, so that the quality of the data is in fact a constant across models. This is different from the previous study (Caucheteux & King, 2022) that reported a left-right asymmetry using a single model. To investigate this issue further, we performed new analyses that we intend to include as supplementary figures. They demonstrate that while the average r-scores (across model sizes), is related to the inter-subjects correlations (ISC), this is not the case for the slope in the relationship between brain correlation and model size.\n\nPlots from Fig.1 in the rebuttal pdf display the left and right, model-free, inter-subjects correlations (ISC)  in various brain regions from the Harvard-Oxford atlas. Some regions, especially in the language network, indeed show asymmetries. Fig. 2a, based on Fig. B5 of the current paper, shows ROI analyses with the addition of horizontal lines indicating the region’s ISCs. Fig. 2b displays the relationship between the average r-score and the ISC, and Fig. 2c the relationship between the slope of r-scores as a function of the number of parameters, and the ISC. Consider for instance aSTS and BA44: in the aSTS, the inter-subjects reliability is greater in the right hemisphere, and on average so is the r-score of the encoding models, but the slope in the relationship between brain correlation and the size of the models is greater in the left aSTS; conversely, in BA44, although the inter-subjects correlation is higher in the left hemisphere, which is reflected in location of the brain correlation of the models on the y-axis, the slopes are quite similar.\n\n> The L-R difference does not seem qualitatively interesting, because both trends in Fig. 6a are straight lines. [...] If the left-hemisphere line were exponentially increasing instead, that may hint at a qualitatively interesting difference between larger vs smaller models. But instead, we see two straight lines. In fact, the L-R difference seems to already be present with models with 350M parameters, and the gap is only linearly widening as model size increases.\n\nIt is not clear to us why only nonlinear effects would be interesting. Perhaps the idea is that steps in the relationship between number of parameters and brain scores would reveal the  sudden emergence of new capacities for the network (?). This is not what we observed, and in that sense, it means that we learn something from the data.\n\n> For the growing L-R asymmetry result, there is no significant effect in many key regions of the language network (IFG, temporal pole) (Figures 4 and 7) (Lines 225-227). The strongest effects seem to occur in AG, which some consider not part of the language network [1]. This is a relevant weakness since this paper focuses on the language network.\n\n5 out of the 7 ROIs actually show significant effects (and 2 of them are located in the IFG). Regarding the Angular Gyrus (AG), while it does not often show up in language minus control contrasts and is thus not included in the core language network by some researchers, it has long been known to be sensitive to linguistic operations, for example, semantic composition (Price et al. 2015 cited in our paper). This is the rationale why we included it in the a priori list of ROIs (before looking at the data!). We are not making a strong statement about what regions belong or not to the language network here, but we think that the inclusion of the Angular Gyrus is reasonable.\n\nThe fact that the slopes do not differ in a BA44 is a weakness only if one a priori considers that it should show an asymmetry. The role of BA44 is actually fiercely debated and some authors consider it to be involved in bilateral articulatory/phonological processes (Matchin & Hickok, 2020, Cereb. Cortex). So rather than a weakness, we take the BA44 result, if it is replicated, as an empirical fact to be explained.\n\n> Because of inter-individual variability in the precise anatomical locations and sizes/shapes of functional areas in the language network, any voxel defined in a common anatomical space often corresponds to different functional areas across individuals. Hence, using group averages and anatomical ROI localization can lead to the blurring of neighboring areas and information loss\n\nThis remark is correct. The Little Prince fMRI dataset does not contain a language localizer that would have permitted us to identify the most sensitive voxels at the individual level. However, given the size of our ROIs and the smoothness of the data we are working with, our experience is that selecting individual voxels from a functional localizer has very little impact. Without doubt, ROIs analyses, by averaging over voxels with different functional profiles, blur the results. Actually, any kind of averaging between subjects has this issue. In our opinion, the ROIs analyses are still a worthy complement to a voxel-based view, that we present on Fig.4.\n\n> Hyperlinks to appendix figures do not work correctly. E.g., \"Fig. B.1\" (Line 162), also: Lines 173, etc. These are the hyperlinks that start with \"Fig B.\"\n\nThanks for pointing this out, it is corrected in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to sincerely thank all four reviewers for their detailed and valuable reviews of our manuscript. We think they have helped to clarify our contribution and strengthen our paper by adding some checks, as described below.

An important point, raised by most reviewers, concerns the potential impact of differences in signal-to-noise ratio in the right and left hemispheres. Indeed, in regions where the signal-to-noise ratio is stronger, one intuitively expects higher correlations between brain data and model predictions. Could this explain away our findings?

To address this issue, we conducted new analyses and generated figures in the accompanying pdf. 
Fig.1 provides more details about inter-subjects correlations (model free) in the left and right hemispheres. In a nutshell, the inter-subjects correlation (ISC) follows the diagonal in left vs. right plot as a general trend, but several regions in the left hemisphere do indeed show higher reliability.
Fig. 2 shows that in the different regions of interest the average brain correlation follows the reliability of the signal (ISC), but the slope in the relationship between brain correlation and number of parameters do not generally follow such a trend.
So, differences in signal-to-noise ratio partly explain the mean brain score difference between left and right, but do not explain the increase in the difference. In other words, while a signal-to-noise ratio difference may explain an asymmetry in a single model, it does not account for the fact that larger models fit better and better left hemispheric activations than right hemispheric ones.

In addition, since the original submission, we have had time to extend the work to Chinese and French data from the Le Petit Prince dataset. Even though we found fewer pretrained large language models available for these two target languages, the analyses essentially reproduce the results obtained with English, in particular with respect to the scaling law in the left-right difference in brain correlation. We intend to add these additional results to the camera-ready paper.

Meta-review: This paper presents a novel result about the asymmetry of language representations in natural speech understanding when modeled with language model of increasing complexity. This is an important question because there is a disagreement in the field: traditional analyses always focus on the left hemisphere, while experiments with natural stimuli often report bilateral representations. The authors proceed by testing models of various sizes and replicating the existence of a scaling law relating the number of model parameters and the ability to predict fMRI activity. They observe that this pattern exhibits a significant Left-Right asymmetry. The reviewers have appreciated the strength of the arguments, especially after adding the additional analysis. The recommendation is to accept this paper as long as the author incorporate the new analyses in the manuscript, along with the appropriate discussion. Is it also particularly valuable to expand on the potential theoretical reasons for this result, since while it does agree with the historical literature on language processing, it is different from what has been recently reported in studies using language models.
Decision: Accept (poster)