Joint processing of linguistic properties in brains and language models
Authors: SUBBA REDDY OOTA
Manish Gupta
Mariya Toneva
Venue: NeurIPS 2023 poster (2023)
Keywords: Linguistic properties
fMRI
probing tasks
cognitive neuroscience
language models
NLP

Abstract: Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available https://github.com/subbareddy248/lingprop-brain-alignment.

Joint processing of linguistic properties in brains and language models
Subba Reddy Ootal 1.2, Manish Gupta Mariya Toneva²
Inria Bordeaux, France, MMPI for Software Systems, Saarbrücken, Germany, *Microsoft, India subba- subba-reddy reapaa-reddyooooaooeeeee ootaainriaaf,, paansshmiccosofee com, mtoneva@mpi-srg org.org
Abstract
Language models have been shown to be very effective in predicting brain record- ings of subjects experiencing complex language stimuli. For deeper understanding this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available
Introduction
Language models that have been pretrained for the next word prediction task using millions of text documents can significantly predict brain recordings of people comprehending language (Wehbe et al. 2014 Jain & Huth 2018: Toneva & Wehbe 2019 Caucheteux & King 2020 Schrimpf et al. 2021 Goldstein et al.] 2022). Understanding the reasons behind the observed similarities between language comprehension machines and brains can lead more insight into both systems.
While such similarities have been observed at a coarse level, it is not yet clear whether and how the two systems align in their information processing pipeline. This pipeline has been studied separately in both systems. In natural language processing (NLP), researchers use probing tasks to uncover the parts of the model that encode specific linguistic propertics (c.g. sentence length. tree depth, top constituents, tense, bigram shift, subject number, object number) (Adi et al.] 2016 Hupkes al. 2018 Conneau et al. 2018 Jawahar al., 2019 Rogers al.) 2020). These techniques have revealed hierarchy of information processing in multi-layered language models that progresses from simple to complex with increased depth. In cognitive neuroscience, traditional experiments study the processing of specific linguistic properties by carefully controlling the experimental stimulus and observing the locations time points of processing in the brain that are affected the most by the controlled stimulus (Hauk & Pulvermüller 20044] Pallier et al., (201T).
More recently, researchers have begun study the alignment of these brain language regions with the layers of language models, and found that the best alignment was achieved in the middle layers of these models (Jain & Huth 2018 Toneva & Wehbe 2019 Caucheteux & King 2020). This has been
https://githin httph///w.w..euuuaae//////////// http///////ngnnoopoooo heciingrroobbbbrinnee brain- al alignment
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ky
Linguistic property
Residual =
f())
Residual encoding performance
Significant difference = Ling. prop. affects alignment
This is Los Angeles.
the
Original encoding performance
Language
Naturalistic stimulus
model
Figure I: Approach to directly test for the effect of a linguistic property on the alignment between language model and brain recordings. First, we remove the linguistic property from the language model representations by learning a simple linear function f that maps the linguistic property to the language model representations, and use this estimated function to obtain the residual language model representation without the contribution of the linguistic property. Next, we compare the brain aligmment (i.e. encoding model performance) before and after the removal of the linguistic property by learning simple linear functions y and / that map the full and residual language model representations to the brain recordings elicited by the corresponding words. Finally, we test whether the differences in brain alignment before and after the removal of the linguistic property are significant and. so, conclude that the respective linguistic property affects the alignment between the language model and brain recordings.
hypothesized to be because the middle layers may contain the most high-level language information as they are farthest from the input and output layers, which contain word-level information due the self-supervised training objective. However, this hypothesis is difficult to reconcile with the results from more recent NL.P probing tasks Conneau al. (2018), Jawahar et al.) (2019), which suggest that the deepest layers in the model should represent the highest-leve language information. Taken together, these findings open up the question of what linguistic information underlies the observed alignment between brains and language models.
Our work aims examine this question via direct approach (see Figure I for a schematic). For number of linguistic properties, we analyze how the alignment between brain recordings and language model representations is affected by the elimination of information related to each linguistic property. For the purposes of this work, we focus on one popular language model-BERR (Devlin) al. which has both been studied extensively in the NLP interpretability literature (i.e.
BER Jawahar et al. (2019)) and has been previously shown to significantly predict fMRI recordings of people processing language (Toneva & Wehbe 2019) [Schrimpf et al., (02T). We test the effect of a range of linguistic properties that have been previously shown to be represented in pretrained BERT (Jawahar et al., 2019). We use a dataset of fMRI recordings that are openly available (Nastase et al. (222)) and correspond to 18 participants listening to natural story.
Using this direct approach, we find that the elimination of each linguistic property results in significant decrease in brain aligmment across all layers of BERT. We additionally find that the syntactic properties (Top Constituents and Tree Depth) have the highest effect on the trend of brain alignment across model layers. Specifically, Top Constituenss is responsible for the bump in brain alignment in the middle layers for all language regions whereas Tree Depth has an impact for temporal (ATL and PTI.) and frontal language regions (IFG and MFG). Performing the same analyses with second popular language model, GPT2 (Radford et 2019), yielded similar results (see Appendix section
Our main contributions are follows:
1. We propose direct approach to evaluate the joint processing of linguistic properties in brains and language models.
2. We show that removing specific linguistic properties leads to a significant decrease in brain alignment. We find that the tested syntactic properties are the most responsible for the trend of brain alignment across BERT layers.
3. Detailed region and sub-region analysis reveal that properties that may not impact the whole brain alignment trend, may play significant role in local trends (c.g., Object Number for

---

ATL and IFGOrb regions, Tense for PCC regions, Sentence Length and Subject Number for PFm sub-region).
We make the code publicly available
2 Related Work
2022). who employ a similar residual Our work is most closely related to that of Toneva et al.
approach study the supra-word meaning of language by removing the contribution of individual words to brain alignment. We build on this approach to study the effect of specific linguistic properties on brain alignment across layers of a language model. Other recent work has examined individual attention heads in BERT and shown that their performance on syntactic tasks correlate with their brain encoding performance in several brain regions (Kumar al., (022). Our work presents complementary approach that gives direct evidence for the importance of a linguistic property to brain alignment.
This work also relates to previous works that investigated the linguistic properties encoded across the layer hierarchy of language models (Adi al., 2016, Hupkes et al.) 2018 Conneau al. 2018 Jawahar et al., 2019) Rogers et al. 2020). Unlike these studies in NLP,, we focus on understanding the degree which various linguistic properties impact the performance of language models in predicting brain recordings.
Our work also relates to growing literature that relates representations of words in language models to those in the brain. A number of studies have related brain responses to word embedding
methods (Pereira et al., 2016 Anderson et al. 2017: Pereira al. 2018 Toneva & Wehbe 2019
Hollenstein et al. 2019; Wang et al. 2020), sentence representation models (Sun et al. [019
Toneva
& Wehbe Sun et al. 2020). recurrent neural networks (Jain & Huth 2018: et al.
2019).
and Transformer methods (Gauthier & Levy 2019) Toneva & Wehbe 2019; Schwwrtz et al. 2019
Jat et al., (2000) Schrimpf et [
et al. 2022 Oota et al. 2022b Merlin & Toneva
2022] Aw & Toneva, 2023 Oota et al., 2023).. Our approach is complementar to these previous
works and can be used further understand the reasons behind the observed brain alignment.
3 Dataset Curation
Brain Imaging Dataset The "Narratives" collection aggregates variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. We analyze the "Narratives-211* year" dataset (Nastase et al. 2021), which is one of the largest publicly available fMRI datasets (in terms of number of samples per participant). The dataset contains data from 18 subjects who listened to the story titled "21st year". The dataset for each subject contains 2226 samples (TRs-Time Repetition). The dataset was already preprocessed and projected on the surface space ("fsaverage6"). We use the multi-modal parcellation of the human cerebral cortex (Glasser Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps (Glasser et al.] 2016). since the Narratives dataset contains annotations that correspond to this atlas. The data covers seven language brain regions of interest. (ROIs) in the human brain with the following subdivisions: (i) angular gyrus (AG: PFm, PGs, PGi, TPOJ2, and TPOJ3); (ii) anterior temporal lobe (ATL: STSda, STSva, STGa, TE1a, TE2a, TGv, and TGd); (iii) posterior temporal lobe (PTL: A4, A5, STSdp, STSvp, PSL, STV, TPOJ1); (iv) inferior frontal gyrus (IFG: 44, 45, IFJa IFSp): (v) middle frontal gyrus (MFG: 55b): (vi) inferior frontal gyrus orbital (IFGOrb: a47r, p47r, a9-46v), (vii) posterior cingulate cortex (PCC: 31 31pd, PCV, 7m, 23, RSC); and (vii) dorsal medial prefrontal cortex (dmPFC: 9m, 10d, d32) (Baker et al. 2018
Milton et al. 2021
Desai et al.
2022
Extracting Word Features from BERT We investigate the alignment with the base pretrained BERT model, provided by Hugging Face (Woff et al. (2220) (12 layers, 768 dimensions). The primary rationale behind our choice to analyze the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-basee
encoders, field often referred to as "BERTology" (Jawahar et al. 2019 Mohebbi et al. 2021). We follow previous work to extract the hidden-state representations from each layer of pretrained BERT, given a fixed-length input (Toneva Wehbe (209).. To extract the stimulus maximum context of previous
features from pretrained BERT, constrained the tokenizer
3

---

Object Number
0.002
0.013
-0.005
-0.104
0.639
1.0
Subject Number
0.047
-0.007
0.067
-0.136
1.0
0.639
6.
Tense
-0.064
0.126
-0.2
1.0
-0.136
-0.104
TopConstituents
0.094
0.133
1.0
-0.2
0.067
-0.005
TreeDepth
-0.138
1.0
0.133
0.126
-0.007
0.013
Sentence Length
1.0
-0.138
0.094
-0.064
0.047
0.002
Sentence Length
TreeDepth
TopConstiuentss
Tense
Subject Number
Object Number
Task Similarity (Pearson Correlation Coefficient) constructed from the busk labels across six tasks. We observe high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.
Figure 2:
20 words. Given the constrained context length, each word successively input to the network with most C previous tokens. For instance, given story of M words and considering the context length of 20, while the third word's 's vector is computed by inputting the network with (W1. W2, Wa). the last word's vectors War is computed by inputting the network with (WM-20:.... .... Wor). The pretrained Transformer model outputs word representations at different encoder layers. We use the #words x 768 dimension vector obtained from each hidden layer to obtaín word-level representations from BERT. For the analyses using GPT-2, we extract the stimulus features in an identical way from a pretrained GPT-2, provided by Hugging Face (Wolf et al. 2020) (12 layers, 768 dimensions).
Downsampling Since the rate of fMRI data acquisition (TR = 1.5sec) was lower than the rate at which the stimulus was presented to the subjects, several words fall under the same TR in a single acquisition. Hence, we match the stimulus acquisition rate to fMRI data recording by downsampling the stimulus features using a 3-lobed Lanczos filter. After downsampling, we obtain the chunk- embedding corresponding to each TR.
TR Alignment To account for the delay of the hemodynamic response, we model the hemodynamic response function using finite response filter (FIR) per voxel and for each subject separately with temporal delays corresponding to 12 seconds (Nishimoto et al. 2011).
Probing Tasks Probing tasks (Adi et al. 2016 Hupkes et al. 2018 Jawahar et al. 2019) help in unpacking the linguistic features possibly encoded neural language models. In this paper, we use six popular probing tasks grouped into three categories- surface (sentence length), syntactic (tree depth and top constituents). and semantic (tense, subject number, and object number)-to assess the capability of BERT layers in encoding each linguistic property. Since other popular probing tasks such as Bigram Shift, Ddd-Man-Oui and Coordination Inversion (as discussed in Conneau et al.) (2018) focused the remaining six probing tasks mentioned
have constant labels for brain dataset, above. Details of each probing task are as follows. Sentence Length: This surface task tests for the length sentence. TreeDepth: This syntactic task tests for classification where the goal is to predict depth of the sentence's ' syntactic tree (with values ranging from 5 to 12). Tontonstituents: This syntactic task tests for the sequence of top-level constituents in the syntactic tree. This is 20-class classification task (e.g. ADVP NOP __P___ CC__ADV NP VP_). Tense: This semantic task tests for the binary classification, based on whether the main verb of the sentence marked as being in the present (PRES class) or past (PAST class) tense. Subject Number: This semantic task tests for the binary classification focusing on the number of the subject of the main clause
sentence. The classes are NN (singular) and NNS (plural or mass: "colors", "waves", etc). Object Number: This semantic task tests for the binary classification focusing on the object number in the main clause of sentence. Details of these tasks are mentioned in Conneau et al. (2018).

---

Balancing Classes As discussed in Section 4 our method involves removing one linguistic property p at time. Ideally, removing p should not impact the performance for any other property p'. However, if some classes of p co-occur very frequently with a few classes of p', this undesired behavior may happen. The Left column of Appendix Figure 6 shows the presence of such frequent class co- occurrences across properties. To avoid skewed class co-occurrences, we regroup the class labels for three tasks: Sentence Length, TreeDepth and TopConstituents, as shown Appendix Figure 5
Probing Task Annotations To annotate the linguistic property labels for each probing task for our dataset, we use Stanford core-NLP stanza library (Manning al., (2014) for sentence-level annotations. Further, for word-level annotations, we assign the same label for all the words present in a sentence except for the sentence length task.
Probing Tasks Similarity Pearson correlation values between task labels for each pair of tasks were used construct the similarity matrix for the "21st year" stimulus text, shown in Figure 2 We observe a high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.
4 Methodology
Removal of Linguistic Properties To remove linguistic property from the pretrained BERT representations, we use ridge regression method in which the probing task label is considered as input and the word features are the target. We compute the residuals by subtracting the predicted feature representations from the actual features resulting in the (linear) removal a linguistic property from pretrained features (see Figure I for a schematic). Because the brain prediction method is also a linear function (see next paragraph), this linear removal limits the contribution of the linguistic property to the eventual brain prediction performance.
Specifically, given an input matrix T; with dimension N x for probing task i, and target word representations W E RNxd, where N denotes the number of words (8267) and d de- notes the dimensionality of each word (768 dimension), the ridge regression objective function is f(T,) = min W - T + x| I] where 0; denotes the learned weight coefficient for em-
bedding dimension d for the input task i, ||-1 denotes the Frobenius norm, and À >> 0 is a tunable hyper-parameter representing the regularization weight for each feature dimension. Using the learned weight coefficients, we compute the residuals as follows: r(T,) W T,,
We verified that another popular method of removing properties from representations Iterattte Null Space Projection (INLP) (Ravfogel al. 2020) leads to similar results (see Appendix Table We present the results from the ridge regression removal in the main paper due to its simplicity.
Similar to the probing experiments with pretrained BERT, we also remove linguistic properties from GPT-2 representations and observe similar results (see Appendix Table IT)
Voxelwise Encoding Model To explore how linguistic properties are encoded in the brain when listening to stories, we use layerwise pretrained BERT features as well as residuals by removing each linguistic property and using them in voxelwise encoding model predict brain responses. If linguistic property is a good predictor of a specific brain region, information about that property is likely encoded in that region. In this paper, we train fMRI encoding models using Banded ridge regression (Tikhonov et al.) 1977) on stimulus representations from the feature spaces mentioned first z-scored each feature channel separately for training and
above. Before doing regression, testing. This was done to match the features to the fMRI responses, which were also z-scored for training and testing. The solution to the banded regression approach is given by f(²) = argmin|| Y x + 1|| where Y denotes the voxels matrix across TRs, B denotes the learned
regression coefficients, and X denotes stimulus or residual representations. regularization parameter for each feature space, we use range of regularization parameters that is explored using cross-validation. The main goal of each fMRI encoding model is predict brain responses associated with each brain voxel given stimulus.
find the optimal
5

---

Table 1: Word-Level Probing task performance for each BERT layer before and after removal of each linguistic property using the 21 year stimuli.
Layers.
Sentence Length
TreeDepih
TopConstinuents
Tense
Subject Number
Object Number
3-classes (Surface)
3-classes
2-classes
2-classes
2-closses
2-classes
(Syntactic)
(Syntactic)
(Somantic)
(Semantic)
(Semantic)
before
after
before
after
before
after
before
after
before
after
before
alter
74.67
43.28
76.30
42.93
77.15
47.28
87.00
59.25
92.10
49.95
93.28
47.31
42.44
76.72
38.88
78.
42.75
87.18
48.2
55.50
93.47
54,59
3 72.31
46.19
75.76
40.33
77.81
48.85
67.42
44,26
93.04
48.55
93.80
49.76
71.34
75.94
38.63
78.36
88.09
42.56
93.50
50.12
94.90
50.06
5
72.67
46.97
76.00
40.88
78.00
45.28
88.39
44.26
94.05
49.88
93.59
51.45
6
70.38
44.37
79.02
41.89
80.23
43.47
87.17
44.44
94.98
55.08
94.50
54.17
7
72.98
46.55
77.93
41.23
80.23
46.43
88.69
42.62
95 95.88
50.24
94.62
47.58
8
72.67
44.67
76.07
40.08
78.90
46.86
87.42
44,56
96.10
50,24
95.10
50.18
70.50
77.15
42.62
52.78
94.56
49.27
10
71.91
47.93
76.90
41.78
78. 17
47.76
88.90 94.994
45.47
96.06
53.68
94,50
50.30
70.07
46.67
77.27
77.65
87.24
48
96.94
94.92
49.52
12
71.77
42.93
76.39
46.61
78.2
78.29
48.67
86.88
45.
94.03
51.45
93.95
48.73
Cross- Validaail data samples from K-1 folds were used for training, and the generalization was tested on samples from the left-out fold.
The ridge regression parameters were fit using 4-fold cross-validation All the
Evaluation Metrics We evaluate our models using Pearson Correlation (PC) which is a popular metric for evaluating brain alignment (Jain & Huth 2018 Schrimpt et al. 202T Goldstein et al. 2022). Let TR be the number of time repetitions. Let Y = {Yi)e and Y = (1))/1 denote the actual and predicted value vectors for single voxel. Thus, Y E RTR and also Ý E RTR We use Pearson Correlation (PC) which is computed as corr(Y, Y) where corr is the correlation function.
Implementation Details for Reproducibility All experiments were conducted on a machine with I NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We used banded ridge-regression with the following parameters: MSE loss function, and L22-eecy (A) varied from 10a to 10"; the best À was chosen by tuning on validation data; the number of cross-validation runs was 4.
5
Results
5.1
Successful removal of linguistic properties from pretrained BERT
To assess the degree to which different layers of pretrained BERT representation encode labels for each probing task, a probing classifier trained over with the embeddings of each layer. For all probing tasks, we use logistic regression as the probing classifier. We train different logistic regression classifiers per layer of BERT using six probing sentence-level datasets created by Conneau et (211)) 8). At test time, for each sample from 21st year, we extract stimuli representations from each layer of BERT, and perform classification for each of the six probing tasks using the learned logistic regression models. To investigate whether a linguistic property was successfully removed from the pretrained representations, we further test whether we can decode the task labels from the residuals for each probing task of the 21 year stimuli.
Table Treports the result for each probing task, before and after removal of the linguistic property from pretrained BERT. We also report similar results for GPT2 in Table I in the Appendix. Similarly to earlier works (Jwaharr et al., 2019). we find that BERT embeds a rich hierarchy of linguistic signals also for our 21 year stimuli: surface information at the initial to middle layers, syntactic information in the middle, semantic information at the middle to top layers. We verify that the removal of each linguistic property from BERT leads to reduced task performance across all layers, as expected.
We further test whether removing information about one property affects the decoding performance for another property. We observed that most properties are not substantially affected by removing other task properties (see Appendix Tables 5 6 and 10).
5.2 Removal of linguistic properties significantly decreases brain alignment across all layers
In Figure (lef)) we present the average brain alignment across all layers of pretrained BERT before and after the removal of cach linguistic property. Removing each linguistic property leads

---

Average across all the lavers
pretrained
Removal TopConstiruents
0.1
0.11
Avg Pearson Correlation
Avg Pearson Correlation
0.1
0.00
0.09
0.08
00.07
Randlan vecttrr
prattiied Rorooval Santerir Lengit
Peymocul of SuyCansttured Eumoval of TreeDrrm
Ramoiia Subject Humber Remorial Teriee
Erpeooved of al Tasto Remmoral of Chject Chyectazzees kazmber
Piemooull of Randon
BERT
0.06
2
8
6
9
10 11 12
Layer Depth
Figure 3: Brain alignment pretrained BERT before (blue) and after (all other bars) removal of different linguistic properties. Left plot compares the average Pearson correlation across all layers of pretrained BERT and all voxels, and the same quantity after removal of each linguistic property. The bars indicate the standard error of the mean across participants. The right plot compares the layer-wise performance of pretrained BERT and removal of one linguistic property-TopConstiing red at particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer. The layer-wise results for removing other linguistic properties differ across properties, but are significant the same layers as TopConstituents and are presented in Appendix Figure
to significant reduction brain alignment. In contrast, removing a randomly generated linguistic property vector (for this baseline, think about new task which has linguistic meaning. The values for each row is then set to a random value chosen uniformly from across all class labels) does not significantly affect the brain alignment. This validates that the reduction in brain alignment observed after removing the linguistic information is indeed due to the elimination of linguistic information, rather than the removal method itself. Further, we observe that the brain alignment of random vector is significantly lower than the residual brain alignment after removal of each linguistic property, This suggests that there are additional important properties for the alignment between pretrained BERT and the fMRI recordings, beyond the ones considered in this work.
In Figure 3 (right), also report the layer-wise performance for pretrained BERT before and after the removal of one representative linguistic SonpertyTTopCoonsitueen We present the results for the remaining properties Figure the Appendix. We also report similar results for GPT2 in Figure in the Appendix. Similarly to previous work (Toneva & Wehbel 2019). we observe that pretrained BERT has the best brain alignment in the middle layers. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure). We observe the most substantial reductions in brain alignment middle late layers. This pattern holds across all tested linguistic properties. These results provide direct evidence that these linguistic properties fact significantly affect the alignment between fMRI recordings and pretrained BERT.
Statistical Significance To estimate the statistical significance of the performance differences (across all tasks), we performed a two-tailed paired-sample t-test on the mean correlation values for the subjects. In all cases, we report p-values across layers. For all layers, the main effect of the two-tailed test was significant for all the tasks with ps 0.05 with confidence 95% Finally, the Henjamni-Hochber False Discovery Rate (FDR) correction for multiple comparisons (Benjamini Hochberg 1995) is used for all tests (appropriate because fMRI data is considered to have positive dependence (Genovese] 2000)). Overall, the p-values confirmed that removing each linguistic property from BERT significantly reduced brain alignment.
Figure in the Appendix displays p-values for the brain alignment of pretrained BERT before and after the removal of each linguistic property across all the layers. We observe that the alignment with the whole brain is significantly reduced across all the layers.
ROI-Level Analysis We further examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprchension (Fedorenko et al. 2010
7

---

Table 2: ROI-level (left) and whole brain (right) correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property.
Tasks
AG
ATI.
PT1.
IFG
IFGOdbb
MPG
PCC
dmPFC
Whole Brain
Sentence Length
0.261
0.264
0.220
0.355
0.129
0.319
0.143
0.100
0.216
TrosDepth
0.365
0.421
0.458
0.442
0.257
0.436
0.109
0.027
0,443
TopConstituents
0.489
0.421
0.464
0.516
0.453
0.463
0.459
0.463
0.451
Tense
0.226
0.283
0.307
0.325
0.345
0.339
0.435
0.122
0.248
Subject Number
0.124
0.201
0.231
0.239
0.285
0.228
0.348
0.237
0.254
Object Number
0.306
0.392
0,342
0.313
0.503
0.335
0.328
0.001
0.263
Table Finer-grained sub-ROI-level correlations across layers between 1) differences in decoding
task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property.
Tasks
AG
ATL
PTL
IFG
PFm
PGI
PGs
STGa
STSda
STSdp
A5
TPO11
PSL
STV
SFL
44
45
FFla
IFSp
Sentence I .nggh
0.386
0.381
0.286
0.258
0.227
0.210
0.176
0.230
0.317
0.246
0.231
0.317
0.372
0.397
0.357
Teoc
0.356
0.461
0.479
0.414
0.515
0.550
0.525
0.469
0.287
0.458
0.418
0.430
0.474
0.439
0.516
TopConstitunn
0.479
0.526
0.454
0.426
0.425
0.477
0.428
0,463
0,413
Tense
0.369
0.365
0.276
0.279
0.238
0.368
0.309
0.340
0.314
0.328
0.329
0.285
0.377
0.321
0.338
Subject Number
0.335
0.134
0.068
0.271
0.270
0.225
0.246
0.158
0.302
0.236
0.256
0.283
0.267
0.271
0.164
Object Number
0.396
0,203
0.298
0.443
0.438
0.370
0.287
0.328
0.279
0.398
0.310
0.319
0.335
0.318
0.321
Fedorenko & Thompson-Schil 2014) and word semantics (Binder et al. 2009). We find that across
language regions, the aligmment is significantly decreased by the removal of the linguistic properties across all layers (see Appendix Figure We further investigate the language sub-regions, such as 44, 45, IFJa, PFS, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Appendix Figure 9)
5.3 Decoding task performance vs brain alignment
While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers. For this purpose, we perform both quantitative and qualitative analyses by measuring the correlation across layers between the differences in decoding task performance from pretrained and residual BERT and the differences in brain alignment of pretrained BERT and residual BERT. A high correlation for a specific linguistic property suggests a strong relationship between the presence of this property across layers in the model and its corresponding effect the layer-wiss brain alignment. Note that this analysis can provide a more direct and stronger claim for the effect of a linguistic property on the brain alignment across layers than alternate analyses, such as computing the correlation between the decoding task performance and the brain alignment across layers only at the level of the pretrained model. We present the results of these analyses for the whole brain and language regions, and language sub-regions in Tables and 3 respectively.
High correlations in the last column of Table 2 indicate that the syntactic NLP tasks TopConstituents and Tree have the strongest effect on the alignment between BERT and the whole brain. At the level of the whole brain, surface level and semantic properties have moderate effect on the trend in brain alignment across layers.
Whole Brain Analysis
ROI-Level Analysis Further, we analyze these correlations for each language brain region sepa- rately, and report the results in the remaining columns of 2 We make the following observations: (I) Both TreeDepth and TopConstitunnts are responsible for the bump brain alignment not just for the entire brain but also for several language ROIs TopConstituents is important across all language ROIs, but most important in IFG which is related to syntax. This result aligns with previous work that has found the inferior frontal regions to be sensitive to syntax (Friederici et al. 20033; Friederici] 2012). (2) Unlike TopConstituents which has strong effect on the alignment with all language ROI. TreeDepth has a strong effect only on alignment with the temporal (ATL and PTL. and frontal regions (IFG and MFG). (3) Although semantic properties are not shown to be responsible for the shape in brain alignment at the whole-brain level, semantic properties such as Tense, Subject Number and Object Number affect the alignment more specifically in PCC. This finding agrees with previous work which suggests the PCC supports word semantics (Binder et al. 2009). (4) The surface property
8

---

L.R
L/R
L/R
Removal of Sentence Length
Removal TreeDepth
Removal TopConstituntts
L/R
L/R
L/R
Removal of Tense
Removal of Subject Number
Removal of Object Number
Figure 4: Voxel-wise correlations across layers between 1) differences in decoding task performance before and after removing each property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. Here L/R denotes the left and right hemispheres.
Sentence Length has a strong effect on the alignment with the IFG and MFG, but not as high as the syntactic properties these language ROIs.
Sub-ROI-Level Analysis Each language brain region is not necessarily homogeneous function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. Thus, we further analyze several important language sub-regions that are thought to exemplify the variety of functionality across some of the broader language regions (Rolls et al. 2022): AG sub-regions (PFm, PGi, PGs), ATL sub-regions (STGa, STSda), PTL sub-regions (STSdp, A5, TPOJ1, PSL, STV, SFL), and IFG sub-regions (44, 45, IFJa, IFSp). We present the correlations for each of these sub-regions separately in Table 3 We make the following observations: (1) Although TopConstituent strongly affect the trend in the brain alignment across layers for the IFG, it strongly affects the trend for only three sub-regions (44, 45, and IFSp). It is possible that other factors affect the trend in alignment with IFJa across layers, this IFG region is involved in many cognitive processes, including working memory for maintaining and updating ongoing information (Rainer et al., 1998) Zanto et al., 2011 Gazzaley & Nobre 2012).
(2) While
TopConstituents was shown to have the strongest effect on the brain alignment with the whole IFG, the alignment with two of the IFG sub-regions (IFSp and IFJa) is more strongly affected by Tree Depth. (3) Subject Number has a medium to strong effect on the alignment with AG sub-region PFm. This implies that the semantic property Subject Number important in at least one sub-region of language ROI. (4) Similarly, we find Object Number to be important for the trend of alignment across layers with most language sub-regions of ATL and PTL. Taken together, these results show that the semantic properties are most responsible for the trend in brain alignment across layers for many language sub-regions of ATL, PTL and AG, while syntactic properties have the highest effect on the alignment with the IFG.
Qualitative Analysis To present the correlations above at an even finer grain, we show them now the voxel-wise level in Figure We make the following qualitative observations: (1) These results confirm that Sentence Length and Subject Number are the least related to the pattern of the brain alignment across layers. (2) The effect of Tree Depth, Top Constituents, and Object Number is more localized to the canonical language regions in the left hemisphere and is more distributed in the right hemisphere. (3) The semantic property Object Number has a much larger effect on the alignment with the left hemisphere. Furthermore, Appendix Figure 12 shows the voxel-wise correlations across layers when removing linguistic properties, instead of each individual one. We observe that removing all linguistic properties leads to very different region-level brain maps compared to removing individual

---

linguistic properties (Figure D and that the remaining correlation across layers after removing all linguistic properties is not substantial in the key language regions.
6 Discussion and Conclusion
We propose a direct approach for evaluating the joint processing of linguistic properties in brains and language models. We show that the removal of a range of linguistic properties from both language models (pretrained BERT and GPT2) leads to a significant decrease in brain alignment across all layers in the language model.
To understand the contribution of each linguistic property to the trend of brain aligmment across layers. we leverage an additional analysis for the whole brain, language regions and language sub-regions: computing the correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. For the whole brain, we find that Tree Depth and Top Constituents are the most responsible for the trend in brain alignment across BERT layers. Specifically, TopConstituents has the largest effect on the trend in brain alignment across BERT layers for all language regions, whereas TreeDepth has a strong effect on a smaller subset of regions that include temporal (ATL and PTL) and frontal language regions (IFG and MFG). Further, although other properties may not have large impact at the whole brain level, they important locally. We find that semantic properties, such as Tense Subject Number, and Object Number affect the alignment with more semantic regions, such PCC (Binder et al. 2009).
One limitation of our removal approach is that any information that is correlated with the removed linguistic property in our dataset will also be removed. This limitation can be alleviated by increasing the dataset size, which is becoming increasingly possible with new releases of publicly available datasets. It is also important to note that while we find that several linguistic properties affect the alignment between fMRI recordings and a pretrained language model, we also observed that there is substantial remaining brain alignment after removal of all linguistic properties. This suggests that our analysis has not accounted for all linguistic properties that are jointly processed. Future work can build on our approach to incorporate additional linguistic properties to fully characterize the joint processing of information between brains and language models. The current study was performed using experimental stimulus in English and language models trained English text. While expect our results to be generalizable at least to languages that are syntactically close to English, this should be explored by future work. Additionally, investigating larger language models beyond the two models used in our study, BERT and GPT-2, can contribute to an even more comprehensive understanding of the reasons for aligament between language models and human brain activity.
The insights gained from our work have implications for Al engineering, neuroscience and model interpretabillty-sonn some in the short-term, others in the long-term. AI engineering: Our work most immediately fits in with the neuro-Al AI research direction that specifically investigates the relationship between representations in the brain and representations learned by powerful neural network models. This direction has gained recent traction, especially in the domain of language, thanks to advancements in language models (Schrimpf et al.) 2021; Goldstein al. 2022). Our work most immediately contributes to this line of research by understanding the reasons for the observed similarity in more depth. Specifically, our work can guide linguistic feature selection, facilitate improved transfer learning, and help in the development of cognitively plausible Al architectures. Computational Modeling in Neuroscience: Researchers have started viewing language models as useful model organisms for human language processing (Toneva 2021) since they implement a language system in way that may be very different from the human brain, but may nonetheless offer insights into the linguistic tasks and computational processes that are sufficient or insufficient to solve them (cCllsseyy 1999 Baroni 2020). Our work enables cognitive neuroscientists to have more control over using language models as model organisms of language processing. Model Interpretability: In the long-term, we hope that our approach can contribute to another line of work that uses brain signals to interpret the information contained by neural network models (Toneva & Wehbe), 2019: Aw & Toneva 2023). We believe that the addition of linguistic features by our approach can further increase the model interpretability enabled by this line of work. Overall, we hope that our approach can be used to better understand the necessary and sufficient properties that lead to significant alignment between brain recordings and language models.
10

---

Acknowledgments
This work was partially funded by the German Research Foundation (DFG) DFG Research Unit FOR 5368, We thank Gabriele Merlin and Camila Kolling for providing valuable feedback on earlier versions of this work.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine- tine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv: 1608. 04207, 2016.
Andrew Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. TACL, 5:17-30, 2017.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Cordell M Baker, Joshua D Burks, Robert G Briggs, Andrew K Conner, Chad A Glenn, Kathleen N Taylor, Goksel Sali, Tressie M McCoy, James D Battiste, Daniel L O' O'Donoghue, et al. A connec- tomic atlas of the human cerebrum chapter 7: the lateral parictal lobe. Operative Neurosurgery, 15(suppL_):sS255--53 ():25)5-5955599 2018.
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society 375(1791):2019000)) 2020.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach multiple testing. Journal of the Royal statistical society: series (Methodological), 57(1):289-300 1995.
Jeffrey Binder, Rutvik H Desai, William W Graves, and Lisa Conant. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):766-2790 2009.
Charlotte Caucheteux and Jean-Rémi King. Language processing in brains and deep neural networks: computational convergence and its limits. BioRxiv, 2020.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single! vector: Probing sentence embeddings for linguistic properties. In ACL 201 8-56th Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 2126-21366 Association for Computational Linguistics, 2018
Rutvik Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system, 2022.
Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.044 2018.
E. Fedorenko, P.-. Hsieh, Bieto-Castanon, Whitfield-Gabrilli, and N. Kanwisher. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Neurophysiology, 104(2):1177-1199) 2010.
Evelina Fedorenko and Sharon L Thompson- Schill. Reworking the language network. Trends in cognitive sciences. 18(3):120-12) 2014.
Angela D Friederici. The cortical language circuit: from auditory perception to sentence comprehen- sion. Trends cognitive sciences, 1665)2262-268, 2012.
Angela D Friederici, Shirley-Ann Rüschemeyer, Anja Hahne, and Christian J Fiebach. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cerebral cortex, 1702) 2003.

---

Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-IJCNIP),, 2019.
Adam Gazzaley and Anna Nobre. Top-down modulation: bridging selective attention and working memory. Trendy in cognitive sciences, 16(2):129-135, 2012.
Christopher Genovese, A bayesian time-course model for functional magnetic resonance imaging data. Journal of the American Statistical Association, 95(451):691-703 2000.
Matthew F Glasser, Timothy Coalson, Emma Robinson, Carl Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):111-178 2016.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369-380, 2022.
Olaf Hauk and Friedemann Pulvermüller. Effects of word length and frequency the human event-related potential. Clinical Neurophysiology, 1 1155);1090-1103 2004.
Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang, Cognival: A framework for cognitive word embedding evaluation. In CoNLL, pp. 538-549, 2019.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fMRI. In Advances in Neural Information Processing Systems, pp. 6628 0088-667,450 2018.
S Jat, Tang, Talukdar, and T' Mitchel. Relating simple sentence representations in deep neural networks and the brain. In ACL., pp. 5137-5154, 2020.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019.
Sreejan Kumar, Theodore Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. bioRxiv, 2022.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp. 55-60, 2014.
Michael McCloskey. Networks and theories: The place of connectionism in cognitive science.
Psychological science, 2(6):387-395, 1991.
Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word-level semantics and prediction. arXiv preprini arXiv:22/2.00596 2022.
Camille K Milton, Vukshitha Dhanaraj, Isabella Young, Hugh M Taylor, Peter J Nicholas, Robert G Briggs, Michael Y Bai, Rannulu D Fonseka, Jorge Hormovas, Yueh-Hsin Lin, et al. Parcellation- based anatomic model of the semantic network. Brain and behavior, 11(4):e02065, 2021.
Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. Exploring the role of bert token representations to explain sentence probing results. The 2021 Conference on Empirical Methods in Natural Language Processing, pp. 792-806. Association for Computational Linguistics, 2021.
12

---

Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Ke- shavarzian, Janice Chen, Christopher J Honey. Yaara Yeshurun, Mor Regev, et al. The "narratives" fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1): 1-22, 2021.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack I. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641-1646 2011.
Subba Reddy Oota, Vijay Rowtula, Manish Gupta, and Raju Bapi. Stepencog: convolutional 1stm autoencoder for near-perfect fmri encoding. In IJCN,, pp. 1-8. IEEE, 2019.
Subba Reddy Oota, Frederic Alcxandre, and Xavier Hinaut. Long-term plausibility of language models and neural dynamics during narrative listening, In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022a.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? NAACL, 2022b.
Subba Reddy Oota, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampud. Syntactic structure processing in the brain while listening. Findings of the Association for Computational Linguistics: ACL, 2023.
Christophe Pallier, Anne- Anne-Dominique Devauchelle, and Stanislas Dehaene. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108(6): 2522 2527, 2011.
Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Decoding of generic mental representations from functional mri data using word embeddings. bioRxiv, pp. 057216, 2016.
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):1-13, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Gregor Rainer, Wael F Asaad, and Earl K Miller. Memory fields of neurons in the primate prefrontal cortex. Proceedings of the National Academy of Sciences, 95(25):15008-150) 1998.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. Association for Computational Linguistics, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866 266, 2020.
Edmund T Rolls, Gustavo Deco, Chu-Chung Huang, and Jianfeng Feng. The human language effective connectome. Neurolmage, pp. 119352, 2022.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 11 1(45),, 2021.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias natural language processing models. Advances in neural information processing systems, 32, 2019.
Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Towards sentence-level brain decoding with distributed representations, In AAAI, pp. 7047-7054, 2019.
13

---

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong, Neural encoding and decoding with distributed sentence representations. IEEE Transactions on Neural Networks and Learning Systems, 32(2):589-603 2020.
Andreï Nikolaevich Tikhonov, Vasilij Ja Arsenin, and Vasilit Arsenin. Solutions of ill-posed problems. V H Winston, 1977.
Mariya Toneva. Bridging Language in Machines with Language in the Brain. PhD thesis, Carnegie Mellon University, 2021.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in Neural Information Processing Systems, 32, 2019.
Mariya Toneva, Tom Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022.
Shaonan Wang. Jiajun Zhang, Haiyan Wang. Nan Lin, and Chengqing Zong. Fine- fine-grained neural decoding with distributed word representations. Information Sciences, 507:256-2723 272 2020.
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context- based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 233-243, 2014.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Transformers: ffate-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Theodore P Zanto, Michael T Rubens, Arul Thangavel, and Adam Gazzaley. Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory. Nature neuroscience, 1465)6656-66,, 2011.
14


Reviews:
[{"review_id": "5mczk0eIUr", "reviewer": "Reviewer_wwF4", "summary": "The authors provide an analysis of correlations between BERT representations and fMRI data when abstracting away different linguistic properties from the BERT representations. They provide an in-depth analysis of the ways in which elimination of word length, tree depth, top constituents, tense, subject number and object number have on the overall and layer-wise alignment of BERT with fMRI data from 18 people. The authors find that removing any of these linguistic properties from the BERT representations reduces fMRI data alignment across all layers in the BERT model. Furthermore, syntactic properties are most relevant for the BERT-brain alignment globally, but semantic properties are highly locally aligned with brain regions generally associated with semantic processing.", "questions": "I find Figure 3 very intriguing. Removing any linguistic property results in a correlation drop from 0.1 to about 0.75. The authors also previously added a correlational analysis indicating that most linguistic properties aren't correlated with each other. Consequently, it would make sense to me that when one removes all linguistic properties, the correlation needs to drop further since different sources of variance have been removed which don't correlate with each other but clearly correlate with the fMRI data. However, even this correlation is at about 0.75. Do you have an explanation why this might be?\n\nPicking up on a notion in the \"Weaknesses\" section: Why doesn't figure 3 already provide us with an answer to the central motivation of section 5.3 (\"we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers.\")? Or did you mean to motivate section 5.3 with \"across brain regions\"?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The overall question and method is relevant to the community. This paper contributes to the cognitive debate on the overall alignment that today's language models have with people's cognitive systems, here concretely their linguistic neural processing of stories. By abstracting away a number of linguistic properties from the BERT representations in a controlled way, the results speak for the type of information encoded in BERT that lead to the reasonably high alignment with fMRI data. The work therefore contributes to a better understanding of BERT representations and the usefulness of investigating them as cognitive systems.\n\nThe authors integrate valuable controls into their study, such as random baselines and correlational analyses of tasks that nicely contextualize the findings.\n\nThe paper is very well-written.", "weaknesses": "To me, section 5.3 (Decoding task performance vs. brain alignment) has a number of interesting observations but I'm missing the overall motivation for this section. The authors write \"While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers.\" (line 265ff). There seems to be a close connection between this question and Figure 3a from the previous section and it would be helpful to emphasize what the previous section lacks. To be clear, I think section 5.3 adds a lot, especially when it comes to the alignment of local instead of global brain regions but I currently can't find a motivation for whom this analysis is most relevant -- does this tell us something about BERT or are there any potential insights for neuroscience? The semantic alignment results seem to be backed up by prior findings in neuroscience. Is the same true for the rest, or if not, what does that mean?\n\nBERT's layer-wise analysis of the correlations with fMRI data are central in the overall framing and centered in the main paper. I think that's rightfully so but I would like to encourage the authors to consider a strategy of substantiating claims that \"BERT embeds a rich hierarchy [with] surface information at the initial and middle layers, syntactic information in the middle to top layers [...]\" (lines 218ff). From inspecting the corresponding results table (Table 1), surface information seem maybe slightly more represented in the first and least in the second layer, but all other layers seem to be fairly evenly distributed. For the syntactic information, it seems uniformly high, and only subject number and object number seem to have a fairly monotonously increasing and decreasing structure that might suggest that it can't simply be due to random variance. Is there any way to estimate the variance of these terms or find a definition for when something is robustly represented more in some than other layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.*\n\n**1. Overall motivation for section 5.3 (Decoding task performance vs brain alignment)**\n\n* While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, the main motivation for Section 5.3 is to understand the contribution of each linguistic property to the trend of brain alignment across layers. \n* This section provides more quantitative evidence and qualitative analyses by measuring the correlation across layers between the differences in decoding task performance from pretrained and residual BERT and the differences in brain alignment of pretrained BERT and residual BERT. \n* We also look at the correlation between the drops in performance in decoding and encoding that are due to the removal of a specific linguistic property. Therefore this evidence is more direct.\n\n* Overall, Section 5.3 helps us understand the contribution of each linguistic property to the trend of brain alignment across layers. It provides multiple neuroscience insights at the whole brain level, ROI level and sub-ROI level of language network.\n\n**2. The layer-wise decoding results are not so precise in terms of what ling property maps to what layer.  Is there any way to estimate the variance of these terms or find a definition for when something is robustly represented more in some than other layers? Can we do more to understand this?**\n\n* As the reviewer points out, we observe that a linguistic property doesn’t precisely map to any particular layer (Fig 7 and Fig 13 in the Appendix). \n* Instead, we focus on the brain alignment trend across all layers and specifically investigate the effect of the linguistic property on this trend, by measuring the correlation between the actual drop in performance resulting from the removal of a linguistic property and the subsequent reduction in brain alignment.\n\n**3. Removing any linguistic property results in a correlation drop from 0.1 to about 0.75. However, even this correlation is at about 0.75 when one removes all linguistic properties. Do you have an explanation why this might be?**\n\n* Fig 3a in the paper reports the average Pearson correlation across all layers of pretrained BERT and all voxels (i.e. whole brain analysis). \n* It is possible that we don't see much difference on average because different linguistic properties may be related to different parts of the network and different brain regions. \n* To investigate this possibility, we created a new brain plot (*Fig 2 in the rebuttal PDF*) by measuring the correlation across layers between voxelwise brain alignments before and after removing all linguistic properties. This shows that removing all properties shows very different region-level brain maps compared to removal of individual linguistic properties (Fig 4 in paper), and not much substantial brain alignment is left in the key language regions after removing all linguistic properties. \n\n* Please check the *rebuttal PDF* at “Common responses”.\n\n"}, {"review_id": "DGPlXUCRIR", "reviewer": "Reviewer_YHJW", "summary": "The paper investigates the relationship between specific linguistic properties and brain alignment across layers of a language model. The authors provide evidence for the importance of a linguistic property to the brain alignment. The focus of the study is to understand the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.\n", "questions": "1. **Choice of Language Model:** Could you elaborate on why you chose to focus only on BERT, an encoder-only language model, for your study? Given that previous research has shown a deeper connection between autoregressive language models and the brain, have you considered including autoregressive language models in your analysis? How do you think the inclusion of such models might affect your findings?\n\n2. **Correlation vs. Causation:** Your study aims to identify the effect of linguistic properties on brain alignment, but it relies on correlational metrics. Could you consider methods from the Causality+NLP literature to produce causal model explanations? How do you think a causal analysis might change your findings or conclusions?\n\n3. **Projection Method:** You use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). Could you consider alternative methods for identifying the effect of linguistic information? How do you think a different method might affect your findings?\n", "limitations": "The authors do recognize some important limitations:\n\n1. **Removal Approach:** They note that their method might unintentionally remove information correlated with the linguistic property they're studying. They suggest using larger datasets to mitigate this.\n\n2. **Unaccounted Linguistic Properties:** They admit that there's still significant brain alignment even after removing all studied linguistic properties, indicating that there might be other linguistic properties at play that they haven't considered.\n\n3. **Single Model Use:** They acknowledge that they've only tested their findings on one language model (BERT) and suggest that future work should test other models.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:** The paper offers an interesting perspective on the link between language properties and brain alignment in language models. It builds on existing work but adds a new dimension by directly showing the importance of specific linguistic properties.\n\n**Quality:** The paper uses a large, public fMRI dataset and word features from the BERT model to get word-level representations. The results are interesting. \n\n**Clarity:** The paper is well-structured and easy to understand. It explains the methods and results clearly, making it accessible to a wide range of readers.\n\n**Significance:** The approach paves a path towards a way to understand how linguistic properties and brain alignment relate. This could help improve language models and give new insights in neuroscience and language processing. \n", "weaknesses": "**Choice of Language Model:** The authors have chosen to focus on BERT, an encoder-only language model, for their study. Given that previous research has shown a deeper connection between autoregressive language models and the brain, the choice of BERT might limit the generalizability of the findings. The authors could consider including autoregressive language models in their analysis to provide a more comprehensive understanding of the relationship between linguistic properties and brain alignment.\n\n**Correlation vs. Causation:** The authors aim to identify the effect of linguistic properties on brain alignment, but they rely on correlational metrics, which do not provide evidence of causation. To address this, the authors could consider methods from the causality+NLP literature to produce causal model explanations. This would strengthen their findings and provide more definitive evidence of the impact of linguistic properties on brain alignment. As it currently stands, it’s unclear which conclusions can be drawn from this study.\n\n**Projection Method:** The authors use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). The authors could consider alternative methods for identifying the effect of linguistic information to ensure the accuracy and reliability of their findings. Some potential approaches include causal abstractions, which aim to provide a low-dimensional causal representation of LLMs.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank the reviewer for their positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.*\n\n**1. Choice of Language Model: BERT vs autoregressive models (GPT2)**\n\n* Please check “Common responses”, and Table 1 and Fig 1 in the rebuttal PDF.\n* The primary rationale behind our choice to incorporate the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as \"BERTology\" [1] (as highlighted by Jawahar et al., 2019, and Mohebbi et al., 2021). We will clarify this. \n* We also additionally reported six probing tasks results for GPT2 before and after removal in the rebuttal PDF. \n\n[1] Rogers A, Kovaleva O, Rumshisky A. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics. 2021 Jan 1;8:842-66.\n\n**2. Correlation vs. Causation**\n\n* Thanks for the suggestion. Exploring the distinction between correlation and causation is an interesting avenue for future research work, and we will add it to the discussion. \n* We still believe that our work is a substantial advancement in this area as it allows us to perturb linguistic representations from models and observe the outcome of this perturbation on brain alignment.\n\n**3. Use of Projection Method to identify the effect of linguistic information**\n\n* We have used two methods (residual method and INLP method) to remove information related to linguistic properties from the model representations, and we obtained similar results with both techniques. \n* We use a linear projection method because the brain alignment function is also a linear function. \n* We understand the concerns raised in previous work that more powerful decoding functions may be able to reconstruct some information about the linguistic property, but what we care about is the linear information, since that is what is used to predict the brain response.\n\n* Additionally, our work is most closely related to that of [2], who employ a similar residual approach to study the supra-word meaning of language by removing the contribution of individual words to brain alignment.\n\n[2] Mariya Toneva, Tom M. Mitchell, Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition, Nature Computational Science 2022.\n\n\n\n"}, {"review_id": "s0k6OqR1DX", "reviewer": "Reviewer_9Zaj", "summary": "This paper aims to investigate the correspondence between HIPSs and models based on Neural Networks such as the transformers. The key innovation of this paper is to study this correspondence by elimiating specicific linguistic properties form BERT and to observe how this intervation affects the alignment with fMRI brain recording.", "questions": "see weaknesses", "limitations": "not applicable", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "\n- This is really an innovative standpoint", "weaknesses": "- There is a major innovation of this paper that is the cornerstone of the theory. Yet, this is not listed as a major result: the removal of linguistic properties from Transformers. Indeed, I was expecting citations to other papers as this is not listed in the main results of the paper. Yet, there are not citations and it seems to be proposed in this paper. \n \n\n- Since the removal of linguistic properties from Transformers is a major cornerstone, all the theory could crumble if it is not investigated in a deep and correct way.\n\n- Experiments to demonstrate that the removal of linguistic properties from Transformers are inconclusive. Indeed, Table 1 is not sufficient. It only says that after the adpatation BERT weights are so mess up that the classification task cannot be performed. There should be a comparative part or a sort of proof that the model is not compromised in the other parts. For example, there is a missing column: BERT with random parameters. Indeed, if the \"after\" models behave as BERT with random inizialization, results are not relevant.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.\n\n**1. There is a major innovation of this paper that is the cornerstone of the theory. Yet, this is not listed as a major result: the removal of linguistic properties from Transformers. Indeed, I was expecting citations to other papers as this is not listed in the main results of the paper. Yet, there are not citations and it seems to be proposed in this paper.**\n\n* For the removal of linguistic property information from the model representations, we use two previously published approaches: residual method [1] and INLP method [2].\n\n* These approaches have been peer reviewed in good venues (Nature Computational Science and AAAI). We will clarify this. \n* We also conducted validation experiments to ensure that these removal techniques work in our setting (Please see the Appendix Tables 5-10). \n\n[1] Mariya Toneva, Tom M. Mitchell, Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition, Nature Computational Science 2022.\n\n[2] Zhang X, Wang S, Lin N, Zhang J, Zong C. Probing word syntactic representations in the brain by a feature elimination method. In AAAI 2022..\n\n\n**2. Since the removal of linguistic properties from Transformers is a major cornerstone, all the theory could crumble if it is not investigated in a deep and correct way.**\n\n* To investigate whether a linguistic property was successfully removed from the pretrained representations, we further test whether we can decode the task labels from the residuals for each probing task of the 21st year stimuli. \n* To ensure that removal of a specific linguistic property works well, we show results in Table 1 (of the paper) by decoding from BERT before and after removal, decoding of linguistic property B when removing linguistic property A. \n* We observed that most properties are not substantially affected by removing other task properties. \n* Further, we observe that the brain alignment of a random vector is significantly lower than the residual brain alignment after removal of each linguistic property.\n\n**3. Proof that the model is not compromised in the other parts. For example, there is a missing column: BERT with random parameters.**\n\n* Fig 3 (in the paper) shows random embeddings are way worse compared to any of the “after” models. Table 1 (in the paper) and Appendix Tables 5-10 show that removing property A does not destroy the decoding performance from BERT on property B so therefore the embedding still contains important information.\n* Based on the reviewers’ suggestion, we now perform brain encoding experiments with random weights of BERT. Please check the *Fig 3 in rebuttal PDF* at “Common responses”.\n* Fig 3 (*in the rebuttal PDF*) shows brain predictivity performance using BERT with random weights is significantly worse compared to any of the “after” removal of linguistic property from pretrained BERT model. This shows that residual representations (i.e. removing a linguistic property from pretrained BERT) are informative and meaningful compared to random weights of BERT.\n\n\n"}, {"review_id": "lOch96PCbt", "reviewer": "Reviewer_p9mD", "summary": "While several papers have recently shown that large language model embeddings are aligned with / predictive of fMRI reading data, this paper pushes these analyses forward by asking: what properties of the representations are responsible for this alignment?  In particular, they compare LLM/fMRI alignment on reading data using BERT, but then compare this to BERT's representations _when certain linguistic properties have been removed_ (using both a novel and an existing removal method).  The results show a significant decrease in alignment (crucially, not found when removing \"random\" information or when using random vectors) across all linguistic properties, suggesting that it really is BERT's encoding of linguistic information that's responsible for the alignment with fMRI reading data.  This represents a finer-grained level of analysis than existing alignment works and should be of interest to many in the field.", "questions": "- Table 1: what are baselines for each task?  What would a most-frequent-class baseline score on each?  This would be a helpful comparison for the post-removal numbers.\n- Word length probing: I'm a bit confused here.  Probing for word length should be hard for BERT since it does not have character-level information but instead uses sub-word tokenization.  I noticed in the Appendix that the task is referred to as \"Sentence Length\".  Is the task length-of-individual-word (that's what \"Word Length\" sounds like to me) or is it number-of-words-in-sentence (\"Sentence Length\")?  Consistency with naming, and clarity on the exact task, would be welcome here.\n- Why the choice of BERT as the model for comparison?  In particular, I am curious about the choice of a \"bidirectional\" masked language model as opposed to a \"left-to-right\" language model like the GPT series.  Some would argue that the latter more closely resemble the reading process, and so might also better predict data from reading.\n\n\n### Typographic comments:\n\n- Line 125 needs a closing period.\n- Lines 185 and 186: \"z-scored\" -> \"$z$-scored\".  Same for \"t-test\" and \"p-value\" throughout the paper.", "limitations": "- Use of stanza corenlp for linguistic annotations; these are \"silver\", not \"gold\".\n- Only looked at one model, with relatively weak overall correlations with brain activity.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Analyzes LLM representations' ability to predict fMRI data before and after \"removing\" linguistic information from said representations.\n- Conducts analyses at both macro (whole-model, and whole-brain) as well as micro (layer-wise, ROI and sub-ROI) levels.", "weaknesses": "- The Pearson correlations in Figure 3 are quite low, on average less than 0.1.  What do the authors make of this?  The ROI-level analyses seem relevant, but are not done with the same metric, so direct comparison is hard.\n- Linguistic annotations for the text in the reading data come from an off-the-shelf NLP tool (stanza), so these are \"silver\" rather than \"gold\" annotations.\n- More baselines would have been useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.*\n\n**1. The Pearson correlations in Figure 3 are quite low, on average less than 0.1. What do the authors make of this? The ROI-level analyses seem relevant, but are not done with the same metric, so direct comparison is hard.**\n\n* The Pearson correlation results reported in our work are in similar ranges as reported in previous studies. \n* The correlations are low but significant. \n* We performed a two-tailed hypothesis test for all the tasks and did the Benjamni-Hochberg False Discovery Rate (FDR) correction for multiple comparisons. \n* One primary reason for the low correlation is partially due to noise in the brain recordings. If one estimates a noise ceiling, we see that these numbers correspond to about 70% of the explainable variance by a model representation.\n\n**2. Linguistic annotations for the text in the reading data come from an off-the-shelf NLP tool (stanza), so these are \"silver\" rather than \"gold\" annotations.**\n\n* We agree that the annotations are silver but since current NLP tools show high accuracies, we can assume them to be fairly accurate as done in many other related papers [1], [2].\n\n[1] Ganesh Jawahar, Benoît Sagot, Djamé Seddah. What does BERT learn about structure of language, ACL-2019\n\n[2[ Hosein Mohebbi, Ali Modarressi, Mohammad Taher Pilehvar. Exploring the Role of BERT Token Representations to Explain Sentence Probing Results, EMNLP-2021\n\n\n**3. Table 1: what are baselines for each task? What would a most-frequent-class baseline score on each? This would be a helpful comparison for the post-removal numbers**\n\n* Unfortunately, there is no previous work to compare with. We reported the chance performance of each probing task in Table 4 in Appendix. \n\n* It shows that after removal of a linguistic property our decoding task accuracy is close to chance performance showing that our property removal method works as expected. Also, Figure 5 displays the distribution of class labels in each of the probing tasks. \n\n**4. Word length vs Sent Length**\n\n* It is sentence length, thanks for pointing out the discrepancy. We will correct this.\n\n**5. Why the choice of BERT as the model for comparison over GPT2**\n\n* Please check “Common responses”, and Table 1 and Fig 1 in rebuttal PDF. \n* The primary rationale behind our choice to incorporate the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as \"BERTology\" [3] (as highlighted by Jawahar et al., 2019, and Mohebbi et al., 2021). We will clarify this.\n\n[3] Rogers A, Kovaleva O, Rumshisky A. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics. 2021 Jan 1;8:842-66.\n\n**6. Typos**\n\n* We will address these typos.\n\n\n\n"}]

Author rebuttal:
*We thank the reviewers for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.*

**Why choose BERT over other models? (reviewers YHJW and p9mD)**

* The primary rationale behind our choice to incorporate the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as "BERTology" [1] (as highlighted by Jawahar et al., 2019, and Mohebbi et al., 2021). We will clarify this.

[1] Rogers A, Kovaleva O, Rumshisky A. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics. 2021 Jan 1;8:842-66.

**Results for a GPT2 model (reviewer YHJW)**

* Based on the reviewers’ suggestion, we now perform experiments with a new model - GPT2. We find the results to be similar to BERT (i.e. a rich hierarchy of linguistic signals: initial to middle layers encode surface information, middle layers encode syntax, middle to top layers encode semantics.)
* Table 1 in rebuttal PDF reports the result for each probing task, before and after removal of the linguistic property from pretrained GPT2. We verify that the removal of each linguistic property from GPT2 leads to reduced task performance across all layers, as expected.
* We also report the layer-wise performance for pretrained GPT2 before and after the removal of one representative linguistic property–TopConstituents in Fig 1 of rebuttal PDF. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure). We are working on completing the results for other linguistic properties, but don’t expect them to differ too much from those based on BERT, given that the linguistic properties map onto similar layers (Table 1 in rebuttal PDF).

**Evidence that representations are not unexpectedly compromised by the removal method: comparison of brain encoding results using BERT with random parameters. (reviewer 9Zaj)** 

* Based on the reviewers’ suggestion, we now perform brain encoding experiments with random weights of BERT. 
  - Fig 3 (in the rebuttal PDF) shows brain predictivity performance using BERT with random weights is significantly worse compared to any of the “after” removal of linguistic property from pretrained BERT model. This shows that residual representations (i.e. removing a linguistic property from pretrained BERT) are informative and meaningful compared to random weights of BERT. 










Meta-review: The paper provides an analysis of correlations between BERT representations and fMRI data when abstracting away different linguistic properties from the BERT representations. The authors provide an in-depth analysis of the ways in which elimination of word length, tree depth, top constituents, tense, subject number and object number have on the overall and layer-wise alignment of BERT with fMRI data from 18 people. They find that removing any of these linguistic properties from the BERT representations reduces fMRI data alignment across all layers in the BERT model. Furthermore, syntactic properties are most relevant for the BERT-brain alignment globally, but semantic properties are highly locally aligned with brain regions generally associated with semantic processing.

This is an interesting paper, presenting novel empirical results, which can inform both the development of LMs and shed light on some aspects of how linguistic information is encoded in the human brain (at least in terms of how it relates to the information learnt by the language model from linguistic data). The paper is well-written, and the analyses are thorough and comprehensive (including appropriate controls etc.) The rebuttal addressed the reviewer's questions and concerns very well, providing motivation for the use of the specific models and removal method, presenting results on GPT-2 and implementing additional baselines which further support their claims. All in all, this is a technically sound, interesting and relevant paper, which would inform and stimulate further research in the community.
Decision: Accept (poster)