<figure><img src="image_1.png" /></figure>

HAL
open science

# Joint processing of linguistic properties in brains and language models

Subba Reddy Oota, Manish Gupta, Mariya Toneva

► To cite this version:

Subba Reddy Oota, Manish Gupta, Mariya Toneva. Joint processing of linguistic properties in brains and language models. NeurIPS 2023 - 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems, Dec 2023, New Orleans, United States. hal-04400635

HAL Id: hal-04400635

https://inria.hal.science/hal-04400635v1

Submitted on 17 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Joint processing of linguistic properties in brains and language models

Subba Reddy Oota<fnref n="1" />,², Manish Gupta³, Mariya Toneva²

¹Inria Bordeaux, France, ²MPI for Software Systems, Saarbrücken, Germany, ³Microsoft, India
subba-reddy.oota@inria.fr, gmanish@microsoft.com, mtoneva@mpi-sws.org

## Abstract

Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available<fnref n="1" />.

## 1 Introduction

Language models that have been pretrained for the next word prediction task using millions of text documents can significantly predict brain recordings of people comprehending language (Wehbe et al., 2014; Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King, 2020; Schrimpf et al., 2021; Goldstein et al., 2022). Understanding the reasons behind the observed similarities between language comprehension in machines and brains can lead to more insight into both systems.

While such similarities have been observed at a coarse level, it is not yet clear whether and how the two systems align in their information processing pipeline. This pipeline has been studied separately in both systems. In natural language processing (NLP), researchers use probing tasks to uncover the parts of the model that encode specific linguistic properties (e.g. sentence length, tree depth, top constituents, tense, bigram shift, subject number, object number) (Adi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018; Jawahar et al., 2019; Rogers et al., 2020). These techniques have revealed a hierarchy of information processing in multi-layered language models that progresses from simple to complex with increased depth. In cognitive neuroscience, traditional experiments study the processing of specific linguistic properties by carefully controlling the experimental stimulus and observing the locations or time points of processing in the brain that are affected the most by the controlled stimulus (Hauk & Pulvermüller, 2004; Pallier et al., 2011).

More recently, researchers have begun to study the alignment of these brain language regions with the layers of language models, and found that the best alignment was achieved in the middle layers of these models (Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King, 2020). This has been

<footnote n="1">¹https://github.com/subbareddy248/linguistic-properties-brain-alignment
37th Conference on Neural Information Processing Systems (NeurIPS 2023).</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: Approach to directly test for the effect of a linguistic property on the alignment between a language model and brain recordings. First, we remove the linguistic property from the language model representations by learning a simple linear function $f$ that maps the linguistic property to the language model representations, and use this estimated function to obtain the residual language model representation without the contribution of the linguistic property. Next, we compare the brain alignment (i.e. encoding model performance) before and after the removal of the linguistic property by learning simple linear functions $g$ and $h$ that map the full and residual language model representations to the brain recordings elicited by the corresponding words. Finally, we test whether the differences in brain alignment before and after the removal of the linguistic property are significant and, if so, conclude that the respective linguistic property affects the alignment between the language model and brain recordings.</figcaption></figure>

hypothesized to be because the middle layers may contain the most high-level language information as they are farthest from the input and output layers, which contain word-level information due to the self-supervised training objective. However, this hypothesis is difficult to reconcile with the results from more recent NLP probing tasks Conneau et al. (2018); Jawahar et al. (2019), which suggest that the deepest layers in the model should represent the highest-level language information. Taken together, these findings open up the question of what linguistic information underlies the observed alignment between brains and language models.

Our work aims to examine this question via a direct approach (see Figure 1 for a schematic). For a number of linguistic properties, we analyze how the alignment between brain recordings and language model representations is affected by the elimination of information related to each linguistic property. For the purposes of this work, we focus on one popular language model–BERT (Devlin et al., 2018)–which has both been studied extensively in the NLP interpretability literature (i.e. BERTology Jawahar et al. (2019)) and has been previously shown to significantly predict fMRI recordings of people processing language (Toneva & Wehbe, 2019; Schrimpf et al., 2021). We test the effect of a range of linguistic properties that have been previously shown to be represented in pretrained BERT (Jawahar et al., 2019). We use a dataset of fMRI recordings that are openly available (Nastase et al., 2021) and correspond to 18 participants listening to a natural story.

Using this direct approach, we find that the elimination of each linguistic property results in a significant decrease in brain alignment across all layers of BERT. We additionally find that the syntactic properties (Top Constituents and Tree Depth) have the highest effect on the trend of brain alignment across model layers. Specifically, Top Constituents is responsible for the bump in brain alignment in the middle layers for all language regions whereas Tree Depth has an impact for temporal (ATL and PTL) and frontal language regions (IFG and MFG). Performing the same analyses with a second popular language model, GPT2 (Radford et al., 2019), yielded similar results (see Appendix section F).

Our main contributions are as follows:

1. We propose a direct approach to evaluate the joint processing of linguistic properties in brains and language models.

2. We show that removing specific linguistic properties leads to a significant decrease in brain alignment. We find that the tested syntactic properties are the most responsible for the trend of brain alignment across BERT layers.

3. Detailed region and sub-region analysis reveal that properties that may not impact the whole brain alignment trend, may play a significant role in local trends (e.g., Object Number for
ATL and IFGOrb regions, Tense for PCC regions, Sentence Length and Subject Number for PFm sub-region).

We make the code publicly available¹.

## 2 Related Work

Our work is most closely related to that of Toneva et al. (2022), who employ a similar residual approach to study the supra-word meaning of language by removing the contribution of individual words to brain alignment. We build on this approach to study the effect of specific linguistic properties on brain alignment across layers of a language model. Other recent work has examined individual attention heads in BERT and shown that their performance on syntactic tasks correlate with their brain encoding performance in several brain regions (Kumar et al., 2022). Our work presents a complementary approach that gives direct evidence for the importance of a linguistic property to brain alignment.

This work also relates to previous works that investigated the linguistic properties encoded across the layer hierarchy of language models (Adi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018; Jawahar et al., 2019; Rogers et al., 2020). Unlike these studies in NLP, we focus on understanding the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.

Our work also relates to a growing literature that relates representations of words in language models to those in the brain. A number of studies have related brain responses to word embedding methods (Pereira et al., 2016; Anderson et al., 2017; Pereira et al., 2018; Toneva & Wehbe, 2019; Hollenstein et al., 2019; Wang et al., 2020), sentence representation models (Sun et al., 2019; Toneva & Wehbe, 2019; Sun et al., 2020), recurrent neural networks (Jain & Huth, 2018; Oota et al., 2019), and Transformer methods (Gauthier & Levy, 2019; Toneva & Wehbe, 2019; Schwartz et al., 2019; Jat et al., 2020; Schrimpf et al., 2021; Goldstein et al., 2022; Oota et al., 2022b,a; Merlin & Toneva, 2022; Aw & Toneva, 2023; Oota et al., 2023). Our approach is complementary to these previous works and can be used to further understand the reasons behind the observed brain alignment.

## 3 Dataset Curation

**Brain Imaging Dataset** The “Narratives” collection aggregates a variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. We analyze the “Narratives-21st year” dataset (Nastase et al., 2021), which is one of the largest publicly available fMRI datasets (in terms of number of samples per participant). The dataset contains data from 18 subjects who listened to the story titled “21st year”. The dataset for each subject contains 2226 samples (TRs-Time Repetition). The dataset was already preprocessed and projected on the surface space (“fsaverage6”). We use the multi-modal parcellation of the human cerebral cortex (Glasser Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps (Glasser et al., 2016), since the Narratives dataset contains annotations that correspond to this atlas. The data covers seven language brain regions of interest (ROIs) in the human brain with the following subdivisions: (i) angular gyrus (AG: PFm, PGs, PGi, TPOJ2, and TPOJ3); (ii) anterior temporal lobe (ATL: STSda, STSva, STGa, TE1a, TE2a, TGv, and TGd); (iii) posterior temporal lobe (PTL: A4, A5, STSdp, STSvp, PSL, STV, TPOJ1); (iv) inferior frontal gyrus (IFG: 44, 45, IFJa, IFSp); (v) middle frontal gyrus (MFG: 55b); (vi) inferior frontal gyrus orbital (IFGOrb: a47r, p47r, a9-46v), (vii) posterior cingulate cortex (PCC: 31pv, 31pd, PCV, 7m, 23, RSC); and (viii) dorsal medial prefrontal cortex (dmPFC: 9m, 10d, d32) (Baker et al., 2018; Milton et al., 2021; Desai et al., 2022).

**Extracting Word Features from BERT** We investigate the alignment with the base pretrained BERT model, provided by Hugging Face (Wolf et al., 2020) (12 layers, 768 dimensions). The primary rationale behind our choice to analyze the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as “BERTology” (Jawahar et al., 2019; Mohebbi et al., 2021). We follow previous work to extract the hidden-state representations from each layer of pretrained BERT, given a fixed-length input (Toneva & Wehbe, 2019). To extract the stimulus features from pretrained BERT, we constrained the tokenizer to use a maximum context of previous
<figure><img src="image_4.png" /><figcaption>Figure 2: Task Similarity (Pearson Correlation Coefficient) constructed from the task-wise labels across six tasks. We observe a high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.</figcaption></figure>

20 words. Given the constrained context length, each word is successively input to the network with at most *C* previous tokens. For instance, given a story of *M* words and considering the context length of 20, while the third word's vector is computed by inputting the network with (*w*₁, *w*₂, *w*₃), the last word's vectors *w*<sub>*M*</sub> is computed by inputting the network with (*w*<sub>*M*−20</sub>, ..., *w*<sub>*M*</sub>). The pretrained Transformer model outputs word representations at different encoder layers. We use the #words × 768 dimension vector obtained from each hidden layer to obtain word-level representations from BERT. For the analyses using GPT-2, we extract the stimulus features in an identical way from a pretrained GPT-2, provided by Hugging Face (Wolf et al., 2020) (12 layers, 768 dimensions).

**Downsampling** Since the rate of fMRI data acquisition (TR = 1.5sec) was lower than the rate at which the stimulus was presented to the subjects, several words fall under the same TR in a single acquisition. Hence, we match the stimulus acquisition rate to fMRI data recording by downsampling the stimulus features using a 3-lobed Lanczos filter. After downsampling, we obtain the chunk-embedding corresponding to each TR.

**TR Alignment** To account for the delay of the hemodynamic response, we model the hemodynamic response function using a finite response filter (FIR) per voxel and for each subject separately with 8 temporal delays corresponding to 12 seconds (Nishimoto et al., 2011).

**Probing Tasks** Probing tasks (Adi et al., 2016; Hupkes et al., 2018; Jawahar et al., 2019) help in unpacking the linguistic features possibly encoded in neural language models. In this paper, we use six popular probing tasks grouped into three categories–surface (sentence length), syntactic (tree depth and top constituents), and semantic (tense, subject number, and object number)–to assess the capability of BERT layers in encoding each linguistic property. Since other popular probing tasks such as Bigram Shift, Odd-Man-Out, and Coordination Inversion (as discussed in Conneau et al. (2018)) have constant labels for our brain dataset, we focused on the remaining six probing tasks mentioned above. Details of each probing task are as follows.
**Sentence Length:** This surface task tests for the length of a sentence. **TreeDepth:** This syntactic task tests for classification where the goal is to predict depth of the sentence’s syntactic tree (with values ranging from 5 to 12). **TopConstituents:** This syntactic task tests for the sequence of top-level constituents in the syntactic tree. This is a 20-class classification task (e.g. ADVP_NP_NP_, CC_ADVP_NP_VP_). **Tense:** This semantic task tests for the binary classification, based on whether the main verb of the sentence is marked as being in the present (PRES class) or past (PAST class) tense. **Subject Number:** This semantic task tests for the binary classification focusing on the number of the subject of the main clause of a sentence. The classes are NN (singular) and NNS (plural or mass: “colors”, “waves”, etc). **Object Number:** This semantic task tests for the binary classification focusing on the object number in the main clause of a sentence. Details of these tasks are mentioned in Conneau et al. (2018).
**Balancing Classes** As discussed in Section 4, our method involves removing one linguistic property *p* at a time. Ideally, removing *p* should not impact the performance for any other property *p'*. However, if some classes of *p* co-occur very frequently with a few classes of *p'*, this undesired behavior may happen. The Left column of Appendix Figure 6 shows the presence of such frequent class co-occurrences across properties. To avoid skewed class co-occurrences, we regroup the class labels for three tasks: Sentence Length, TreeDepth and TopConstituents, as shown in Appendix Figure 5.

**Probing Task Annotations** To annotate the linguistic property labels for each probing task for our dataset, we use Stanford core-NLP stanza library (Manning et al., 2014) for sentence-level annotations. Further, for word-level annotations, we assign the same label for all the words present in a sentence except for the sentence length task.

**Probing Tasks Similarity** Pearson correlation values between task labels for each pair of tasks were used to construct the similarity matrix for the “21st year” stimulus text, as shown in Figure 2. We observe a high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.

# 4 Methodology

**Removal of Linguistic Properties** To remove a linguistic property from the pretrained BERT representations, we use a ridge regression method in which the probing task label is considered as input and the word features are the target. We compute the residuals by subtracting the predicted feature representations from the actual features resulting in the (linear) removal of a linguistic property from pretrained features (see Figure 1 for a schematic). Because the brain prediction method is also a linear function (see next paragraph), this linear removal limits the contribution of the linguistic property to the eventual brain prediction performance.

Specifically, given an input matrix $T_i$ with dimension $N \times 1$ for probing task $i$, and target word representations $W \in \mathbb{R}^{N \times d}$, where $N$ denotes the number of words (8267) and $d$ denotes the dimensionality of each word (768 dimension), the ridge regression objective function is $f(T_i) = \min_{\theta_i} \|W - T_i\theta_i\|_F^2 + \lambda \|\theta_i\|_F^2$ where $\theta_i$ denotes the learned weight coefficient for embedding dimension $d$ for the input task $i$, $\|\cdot\|_F^2$ denotes the Frobenius norm, and $\lambda > 0$ is a tunable hyper-parameter representing the regularization weight for each feature dimension. Using the learned weight coefficients, we compute the residuals as follows: $r(T_i) = W - T_i\theta_i$.

We verified that another popular method of removing properties from representations–Iterative Null Space Projection (INLP) (Ravfogel et al., 2020)–leads to similar results (see Appendix Table 4). We present the results from the ridge regression removal in the main paper due to its simplicity.

Similar to the probing experiments with pretrained BERT, we also remove linguistic properties from GPT-2 representations and observe similar results (see Appendix Table 11).

**Voxelwise Encoding Model** To explore how linguistic properties are encoded in the brain when listening to stories, we use layerwise pretrained BERT features as well as residuals by removing each linguistic property and using them in a voxelwise encoding model to predict brain responses. If a linguistic property is a good predictor of a specific brain region, information about that property is likely encoded in that region. In this paper, we train fMRI encoding models using Banded ridge regression (Tikhonov et al., 1977) on stimulus representations from the feature spaces mentioned above. Before doing regression, we first z-scored each feature channel separately for training and testing. This was done to match the features to the fMRI responses, which were also z-scored for training and testing. The solution to the banded regression approach is given by $f(\hat{\beta}) = \underset{\beta}{\operatorname{argmin}} \|Y - X\beta\|_F^2 + \lambda \|\beta\|_F^2$, where $Y$ denotes the voxels matrix across TRs, $\beta$ denotes the learned regression coefficients, and $X$ denotes stimulus or residual representations. To find the optimal regularization parameter for each feature space, we use a range of regularization parameters that is explored using cross-validation. The main goal of each fMRI encoding model is to predict brain responses associated with each brain voxel given a stimulus.
<table><caption>Table 1: Word-Level Probing task performance for each BERT layer before and after removal of each linguistic property using the 21st year stimuli.</caption><thead><tr><th rowspan="2">Layers</th><th colspan="2">Sentence Length<br>3-classes<br>(Surface)</th><th colspan="2">TreeDepth<br>3-classes<br>(Syntactic)</th><th colspan="2">TopConstituents<br>2-classes<br>(Syntactic)</th><th colspan="2">Tense<br>2-classes<br>(Semantic)</th><th colspan="2">Subject Number<br>2-classes<br>(Semantic)</th><th colspan="2">Object Number<br>2-classes<br>(Semantic)</th></tr><tr><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th></tr></thead><tbody><tr><td>1</td><td><strong>74.67</strong></td><td>43.28</td><td>76.30</td><td>42.93</td><td>77.15</td><td>47.28</td><td>87.00</td><td>59.25</td><td>92.10</td><td>49.95</td><td>93.28</td><td>47.31</td></tr><tr><td>2</td><td>69.83</td><td>42.44</td><td>76.72</td><td>38.88</td><td>78.60</td><td>42.75</td><td>87.18</td><td>48.25</td><td>92.32</td><td>55.50</td><td>93.47</td><td>54.59</td></tr><tr><td>3</td><td>72.31</td><td>46.19</td><td>75.76</td><td>40.33</td><td>77.81</td><td>48.85</td><td>87.42</td><td>44.26</td><td>93.04</td><td>48.55</td><td>93.80</td><td>49.76</td></tr><tr><td>4</td><td>71.34</td><td>46.43</td><td>75.94</td><td>38.63</td><td>78.36</td><td>48.00</td><td>88.09</td><td>42.56</td><td>93.50</td><td>50.12</td><td>94.90</td><td>50.06</td></tr><tr><td>5</td><td>72.67</td><td>46.97</td><td>76.00</td><td>40.88</td><td>78.60</td><td>45.28</td><td>88.39</td><td>44.26</td><td>94.05</td><td>49.88</td><td>93.59</td><td>51.45</td></tr><tr><td>6</td><td>70.38</td><td>44.37</td><td><strong>79.02</strong></td><td>41.89</td><td><strong>80.23</strong></td><td>43.47</td><td>87.17</td><td>44.44</td><td>94.98</td><td>55.08</td><td>94.50</td><td>54.17</td></tr><tr><td>7</td><td>72.98</td><td>46.55</td><td>77.93</td><td>41.23</td><td><strong>80.23</strong></td><td>46.43</td><td>88.69</td><td>42.62</td><td>95.88</td><td>50.24</td><td>94.62</td><td>47.58</td></tr><tr><td>8</td><td>72.67</td><td>44.67</td><td>76.07</td><td>40.08</td><td>78.90</td><td>46.86</td><td>87.42</td><td>44.56</td><td>96.10</td><td>50.24</td><td><strong>95.10</strong></td><td>50.18</td></tr><tr><td>9</td><td>70.50</td><td>45.28</td><td>77.15</td><td>42.62</td><td>79.87</td><td>44.55</td><td>88.27</td><td>47.22</td><td>96.38</td><td>52.78</td><td>94.56</td><td>49.27</td></tr><tr><td>10</td><td>72.91</td><td>47.93</td><td>76.90</td><td>41.78</td><td>78.17</td><td>47.76</td><td><strong>88.94</strong></td><td>45.47</td><td>96.06</td><td>53.68</td><td>94.50</td><td>50.30</td></tr><tr><td>11</td><td>70.07</td><td>46.67</td><td>77.27</td><td>45.47</td><td>77.69</td><td>45.77</td><td>87.24</td><td>48.43</td><td><strong>96.94</strong></td><td>53.44</td><td>94.92</td><td>49.52</td></tr><tr><td>12</td><td>71.77</td><td>42.93</td><td>76.39</td><td>46.61</td><td>78.29</td><td>48.67</td><td>86.88</td><td>45.10</td><td>94.03</td><td>51.45</td><td>93.95</td><td>48.73</td></tr></tbody></table>

**Cross-Validation** The ridge regression parameters were fit using 4-fold cross-validation. All the data samples from K-1 folds were used for training, and the generalization was tested on samples from the left-out fold.

**Evaluation Metrics** We evaluate our models using Pearson Correlation (PC) which is a popular metric for evaluating brain alignment (Jain & Huth, 2018; Schrimpf et al., 2021; Goldstein et al., 2022). Let TR be the number of time repetitions. Let $Y = \{Y_i\}_{i=1}^{TR}$ and $\hat{Y} = \{\hat{Y}_i\}_{i=1}^{TR}$ denote the actual and predicted value vectors for a single voxel. Thus, $Y \in R^{TR}$ and also $\hat{Y} \in R^{TR}$. We use Pearson Correlation (PC) which is computed as $corr(Y, \hat{Y})$ where $corr$ is the correlation function.

**Implementation Details for Reproducibility** All experiments were conducted on a machine with 1 NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We used banded ridge-regression with the following parameters: MSE loss function, and L2-decay ($\lambda$) varied from $10^{-1}$ to $10^{-3}$; the best $\lambda$ was chosen by tuning on validation data; the number of cross-validation runs was 4.

# 5 Results

## 5.1 Successful removal of linguistic properties from pretrained BERT

To assess the degree to which different layers of pretrained BERT representation encode labels for each probing task, a probing classifier is trained over it with the embeddings of each layer. For all probing tasks, we use logistic regression as the probing classifier. We train different logistic regression classifiers per layer of BERT using six probing sentence-level datasets created by Conneau et al. (2018). At test time, for each sample from 21st year, we extract stimuli representations from each layer of BERT, and perform classification for each of the six probing tasks using the learned logistic regression models. To investigate whether a linguistic property was successfully removed from the pretrained representations, we further test whether we can decode the task labels from the residuals for each probing task of the 21st year stimuli.

Table 1 reports the result for each probing task, before and after removal of the linguistic property from pretrained BERT. We also report similar results for GPT2 in Table 11 in the Appendix. Similarly to earlier works (Jawahar et al., 2019), we find that BERT embeds a rich hierarchy of linguistic signals also for our 21st year stimuli: surface information at the initial to middle layers, syntactic information in the middle, semantic information at the middle to top layers. We verify that the removal of each linguistic property from BERT leads to reduced task performance across all layers, as expected.

We further test whether removing information about one property affects the decoding performance for another property. We observed that most properties are not substantially affected by removing other task properties (see Appendix Tables 5, 6, 7, 8, 9 and 10).

## 5.2 Removal of linguistic properties significantly decreases brain alignment across all layers

In Figure 3 (left), we present the average brain alignment across all layers of pretrained BERT before and after the removal of each linguistic property. Removing each linguistic property leads
<figure><img src="image_5.png" /><figcaption>Figure 3: Brain alignment of pretrained BERT before (blue) and after (all other bars) removal of different linguistic properties. Left plot compares the average Pearson correlation across all layers of pretrained BERT and all voxels, and the same quantity after removal of each linguistic property. The error bars indicate the standard error of the mean across participants. The right plot compares the layer-wise performance of pretrained BERT and removal of one linguistic property—TopConstituents. A red * at a particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer. The layer-wise results for removing other linguistic properties differ across properties, but are significant at the same layers as TopConstituents and are presented in Appendix Figure 7.</figcaption></figure>

to a significant reduction in brain alignment. In contrast, removing a randomly generated linguistic property vector (for this baseline, think about a new task which has no linguistic meaning. The values for each row is then set to a random value chosen uniformly from across all class labels) does not significantly affect the brain alignment. This validates that the reduction in brain alignment observed after removing the linguistic information is indeed due to the elimination of linguistic information, rather than to the removal method itself. Further, we observe that the brain alignment of a random vector is significantly lower than the residual brain alignment after removal of each linguistic property. This suggests that there are additional important properties for the alignment between pretrained BERT and the fMRI recordings, beyond the ones considered in this work.

In Figure 3 (right), we also report the layer-wise performance for pretrained BERT before and after the removal of one representative linguistic property—TopConstituents. We present the results for the remaining properties in Figure 7 in the Appendix. We also report similar results for GPT2 in Figure 11 in the Appendix. Similarly to previous work (Toneva & Wehbe, 2019), we observe that pretrained BERT has the best brain alignment in the middle layers. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure). We observe the most substantial reductions in brain alignment in middle to late layers. This pattern holds across all tested linguistic properties. These results provide direct evidence that these linguistic properties in fact significantly affect the alignment between fMRI recordings and pretrained BERT.

**Statistical Significance** To estimate the statistical significance of the performance differences (across all tasks), we performed a two-tailed paired-sample t-test on the mean correlation values for the subjects. In all cases, we report p-values across layers. For all layers, the main effect of the two-tailed test was significant for all the tasks with $p \le 0.05$ with confidence 95%. Finally, the Benjamini-Hochberg False Discovery Rate (FDR) correction for multiple comparisons (Benjamini & Hochberg, 1995) is used for all tests (appropriate because fMRI data is considered to have positive dependence (Genovese, 2000)). Overall, the p-values confirmed that removing each linguistic property from BERT significantly reduced brain alignment.

Figure 10 in the Appendix displays p-values for the brain alignment of pretrained BERT before and after the removal of each linguistic property across all the layers. We observe that the alignment with the whole brain is significantly reduced across all the layers.

**ROI-Level Analysis** We further examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprehension (Fedorenko et al., 2010;
<table><caption>Table 2: ROI-level (left) and whole brain (right) correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property.</caption><thead><tr><th>Tasks</th><th>AG</th><th>ATL</th><th>PTL</th><th>IFG</th><th>IFGOrb</th><th>MFG</th><th>PCC</th><th>dmPFC</th><th>Whole Brain</th></tr></thead><tbody><tr><td>Sentence Length</td><td>0.261</td><td>0.264</td><td>0.220</td><td>0.355</td><td>0.129</td><td>0.319</td><td>0.143</td><td>0.100</td><td>0.216</td></tr><tr><td>TreeDepth</td><td>0.365</td><td><strong>0.421</strong></td><td><strong>0.458</strong></td><td><strong>0.442</strong></td><td>0.257</td><td><strong>0.436</strong></td><td>0.109</td><td>0.027</td><td><strong>0.443</strong></td></tr><tr><td>TopConstituents</td><td><strong>0.489</strong></td><td><strong>0.421</strong></td><td><strong>0.464</strong></td><td><strong>0.516</strong></td><td><strong>0.453</strong></td><td><strong>0.463</strong></td><td><strong>0.459</strong></td><td><strong>0.463</strong></td><td><strong>0.451</strong></td></tr><tr><td>Tense</td><td>0.226</td><td>0.283</td><td>0.307</td><td>0.325</td><td>0.345</td><td>0.339</td><td><strong>0.435</strong></td><td>0.122</td><td>0.248</td></tr><tr><td>Subject Number</td><td>0.124</td><td>0.201</td><td>0.231</td><td>0.239</td><td>0.285</td><td>0.228</td><td>0.348</td><td>0.237</td><td>0.254</td></tr><tr><td>Object Number</td><td>0.306</td><td><strong>0.392</strong></td><td>0.342</td><td>0.313</td><td><strong>0.503</strong></td><td>0.335</td><td>0.328</td><td>0.001</td><td>0.263</td></tr></tbody></table>

<table><caption>Table 3: Finer-grained sub-ROI-level correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property.</caption><thead><tr><th rowspan="2">Tasks</th><th colspan="3">AG</th><th colspan="2">ATL</th><th colspan="5">PTL</th><th colspan="4">IFG</th></tr><tr><th>PFm</th><th>PGi</th><th>PGs</th><th>STGa</th><th>STSda</th><th>STSdp</th><th>A5</th><th>TPOJI</th><th>PSL</th><th>STV</th><th>SFL</th><th>44</th><th>45</th><th>IFJa</th><th>IFSp</th></tr></thead><tbody><tr><td>Sentence Length</td><td><strong>0.386</strong></td><td><strong>0.381</strong></td><td>0.286</td><td>0.258</td><td>0.227</td><td>0.210</td><td>0.176</td><td>0.230</td><td>0.317</td><td>0.246</td><td>0.231</td><td>0.317</td><td>0.372</td><td><strong>0.397</strong></td><td>0.357</td></tr><tr><td>TreeDepth</td><td>0.356</td><td><strong>0.461</strong></td><td><strong>0.479</strong></td><td><strong>0.414</strong></td><td><strong>0.515</strong></td><td><strong>0.550</strong></td><td><strong>0.525</strong></td><td><strong>0.469</strong></td><td>0.287</td><td><strong>0.458</strong></td><td><strong>0.418</strong></td><td><strong>0.430</strong></td><td><strong>0.474</strong></td><td><strong>0.439</strong></td><td><strong>0.516</strong></td></tr><tr><td>TopConstituents</td><td><strong>0.479</strong></td><td><strong>0.526</strong></td><td><strong>0.454</strong></td><td>0.336</td><td><strong>0.426</strong></td><td><strong>0.425</strong></td><td><strong>0.477</strong></td><td><strong>0.428</strong></td><td><strong>0.463</strong></td><td>0.360</td><td><strong>0.398</strong></td><td><strong>0.531</strong></td><td><strong>0.581</strong></td><td>0.353</td><td><strong>0.413</strong></td></tr><tr><td>Tense</td><td>0.369</td><td>0.365</td><td>0.276</td><td>0.279</td><td>0.238</td><td>0.368</td><td>0.309</td><td>0.340</td><td>0.314</td><td>0.328</td><td>0.329</td><td>0.285</td><td>0.377</td><td>0.321</td><td>0.338</td></tr><tr><td>Subject Number</td><td>0.335</td><td>0.134</td><td>0.068</td><td>0.271</td><td>0.270</td><td>0.225</td><td>0.246</td><td>0.158</td><td>0.302</td><td>0.226</td><td>0.256</td><td>0.283</td><td>0.267</td><td>0.271</td><td>0.164</td></tr><tr><td>Object Number</td><td><strong>0.396</strong></td><td>0.203</td><td>0.298</td><td><strong>0.443</strong></td><td><strong>0.438</strong></td><td>0.370</td><td>0.287</td><td>0.328</td><td>0.279</td><td><strong>0.398</strong></td><td>0.310</td><td>0.319</td><td>0.335</td><td>0.318</td><td>0.321</td></tr></tbody></table>

Fedorenko & Thompson-Schill, 2014) and word semantics (Binder et al., 2009). We find that across language regions, the alignment is significantly decreased by the removal of the linguistic properties across all layers (see Appendix Figure 8). We further investigate the language sub-regions, such as 44, 45, IFJa, IFSp, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Appendix Figure 9).

## 5.3 Decoding task performance vs brain alignment

While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers. For this purpose, we perform both quantitative and qualitative analyses by measuring the correlation across layers between the differences in decoding task performance from pretrained and residual BERT and the differences in brain alignment of pretrained BERT and residual BERT. A high correlation for a specific linguistic property suggests a strong relationship between the presence of this property across layers in the model and its corresponding effect on the layer-wise brain alignment. Note that this analysis can provide a more direct and stronger claim for the effect of a linguistic property on the brain alignment across layers than alternate analyses, such as computing the correlation between the decoding task performance and the brain alignment across layers only at the level of the pretrained model. We present the results of these analyses for the whole brain and language regions, and language sub-regions in Tables 2 and 3, respectively.

**Whole Brain Analysis** High correlations in the last column of Table 2 indicate that the syntactic NLP tasks TopConstituents and TreeDepth have the strongest effect on the alignment between BERT and the whole brain. At the level of the whole brain, surface level and semantic properties have a moderate effect on the trend in brain alignment across layers.

**ROI-Level Analysis** Further, we analyze these correlations for each language brain region separately, and report the results in the remaining columns of Table 2. We make the following observations: (1) Both TreeDepth and TopConstituents are responsible for the bump in brain alignment not just for the entire brain but also for several language ROIs. TopConstituents is important across all language ROIs, but most important in IFG which is related to syntax. This result aligns with previous work that has found the inferior frontal regions to be sensitive to syntax (Friederici et al., 2003; Friederici, 2012). (2) Unlike TopConstituents which has a strong effect on the alignment with all language ROI, TreeDepth has a strong effect only on alignment with the temporal (ATL and PTL) and frontal regions (IFG and MFG). (3) Although semantic properties are not shown to be responsible for the shape in brain alignment at the whole-brain level, semantic properties such as Tense, Subject Number and Object Number affect the alignment more specifically in PCC. This finding agrees with previous work which suggests the PCC supports word semantics (Binder et al., 2009). (4) The surface property
<figure><img src="image_6.png" /><figcaption>Figure 4: Voxel-wise correlations across layers between 1) differences in decoding task performance before and after removing each property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. Here L/R denotes the left and right hemispheres.</figcaption></figure>

Sentence Length has a strong effect on the alignment with the IFG and MFG, but not as high as the syntactic properties in these language ROIs.

**Sub-ROI-Level Analysis** Each language brain region is not necessarily homogeneous in function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. Thus, we further analyze several important language sub-regions that are thought to exemplify the variety of functionality across some of the broader language regions (Rolls et al., 2022): AG sub-regions (PFm, PGi, PGs), ATL sub-regions (STGa, STSda), PTL sub-regions (STSdp, A5, TPOJ1, PSL, STV, SFL), and IFG sub-regions (44, 45, IFJa, IFSp). We present the correlations for each of these sub-regions separately in Table 3. We make the following observations: (1) Although TopConstituents strongly affect the trend in the brain alignment across layers for the IFG, it strongly affects the trend for only three sub-regions (44, 45, and IFSp). It is possible that other factors affect the trend in alignment with IFJa across layers, as this IFG region is involved in many cognitive processes, including working memory for maintaining and updating ongoing information (Rainer et al., 1998; Zanto et al., 2011; Gazzaley & Nobre, 2012). (2) While TopConstituents was shown to have the strongest effect on the brain alignment with the whole IFG, the alignment with two of the IFG sub-regions (IFSp and IFJa) is more strongly affected by Tree Depth. (3) Subject Number has a medium to strong effect on the alignment with AG sub-region PFm. This implies that the semantic property Subject Number is important in at least one sub-region of language ROI. (4) Similarly, we find Object Number to be important for the trend of alignment across layers with most language sub-regions of ATL and PTL. Taken together, these results show that the semantic properties are most responsible for the trend in brain alignment across layers for many language sub-regions of ATL, PTL and AG, while syntactic properties have the highest effect on the alignment with the IFG.

**Qualitative Analysis** To present the correlations above at an even finer grain, we show them now at the voxel-wise level in Figure 4. We make the following qualitative observations: (1) These results confirm that Sentence Length and Subject Number are the least related to the pattern of the brain alignment across layers. (2) The effect of Tree Depth, Top Constituents, and Object Number is more localized to the canonical language regions in the left hemisphere and is more distributed in the right hemisphere. (3) The semantic property Object Number has a much larger effect on the alignment with the left hemisphere. Furthermore, Appendix Figure 12 shows the voxel-wise correlations across layers when removing all linguistic properties, instead of each individual one. We observe that removing all linguistic properties leads to very different region-level brain maps compared to removing individual
linguistic properties (Figure 4), and that the remaining correlation across layers after removing all
linguistic properties is not substantial in the key language regions.

# 6 Discussion and Conclusion

We propose a direct approach for evaluating the joint processing of linguistic properties in brains and language models. We show that the removal of a range of linguistic properties from both language models (pretrained BERT and GPT2) leads to a significant decrease in brain alignment across all layers in the language model.

To understand the contribution of each linguistic property to the trend of brain alignment across layers, we leverage an additional analysis for the whole brain, language regions and language sub-regions: computing the correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. For the whole brain, we find that Tree Depth and Top Constituents are the most responsible for the trend in brain alignment across BERT layers. Specifically, TopConstituents has the largest effect on the trend in brain alignment across BERT layers for all language regions, whereas TreeDepth has a strong effect on a smaller subset of regions that include temporal (ATL and PTL) and frontal language regions (IFG and MFG). Further, although other properties may not have a large impact at the whole brain level, they are important locally. We find that semantic properties, such as Tense, Subject Number, and Object Number affect the alignment with more semantic regions, such as PCC (Binder et al., 2009).

One limitation of our removal approach is that any information that is correlated with the removed linguistic property in our dataset will also be removed. This limitation can be alleviated by increasing the dataset size, which is becoming increasingly possible with new releases of publicly available datasets. It is also important to note that while we find that several linguistic properties affect the alignment between fMRI recordings and a pretrained language model, we also observed that there is substantial remaining brain alignment after removal of all linguistic properties. This suggests that our analysis has not accounted for all linguistic properties that are jointly processed. Future work can build on our approach to incorporate additional linguistic properties to fully characterize the joint processing of information between brains and language models. The current study was performed using experimental stimulus in English and language models trained on English text. While we expect our results to be generalizable at least to languages that are syntactically close to English, this should be explored by future work. Additionally, investigating larger language models beyond the two models used in our study, BERT and GPT-2, can contribute to an even more comprehensive understanding of the reasons for alignment between language models and human brain activity.

The insights gained from our work have implications for AI engineering, neuroscience and model interpretability—some in the short-term, others in the long-term. **AI engineering:** Our work most immediately fits in with the neuro-AI research direction that specifically investigates the relationship between representations in the brain and representations learned by powerful neural network models. This direction has gained recent traction, especially in the domain of language, thanks to advancements in language models (Schrimpf et al., 2021; Goldstein et al., 2022). Our work most immediately contributes to this line of research by understanding the reasons for the observed similarity in more depth. Specifically, our work can guide linguistic feature selection, facilitate improved transfer learning, and help in the development of cognitively plausible AI architectures. **Computational Modeling in Neuroscience:** Researchers have started viewing language models as useful *model organisms* for human language processing (Toneva, 2021) since they implement a language system in a way that may be very different from the human brain, but may nonetheless offer insights into the linguistic tasks and computational processes that are sufficient or insufficient to solve them (McCloskey, 1991; Baroni, 2020). Our work enables cognitive neuroscientists to have more control over using language models as model organisms of language processing. **Model Interpretability:** In the long-term, we hope that our approach can contribute to another line of work that uses brain signals to interpret the information contained by neural network models (Toneva & Wehbe, 2019; Aw & Toneva, 2023). We believe that the addition of linguistic features by our approach can further increase the model interpretability enabled by this line of work. Overall, we hope that our approach can be used to better understand the necessary and sufficient properties that lead to significant alignment between brain recordings and language models.
## References

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. *arXiv preprint arXiv:1608.04207*, 2016.

Andrew J Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. *TACL*, 5:17–30, 2017.

Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In *The Eleventh International Conference on Learning Representations*, 2023.

Cordell M Baker, Joshua D Burks, Robert G Briggs, Andrew K Conner, Chad A Glenn, Kathleen N Taylor, Goksel Sali, Tressie M McCoy, James D Battiste, Daniel L O'Donoghue, et al. A connec-tomic atlas of the human cerebrum—chapter 7: the lateral parietal lobe. *Operative Neurosurgery*, 15(suppl_1):S295–S349, 2018.

Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. *Philosophical Transactions of the Royal Society B*, 375(1791):20190307, 2020.

Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. *Journal of the Royal statistical society: series B (Methodological)*, 57(1):289–300, 1995.

Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. *Cerebral cortex*, 19(12):2767–2796, 2009.

Charlotte Caucheteux and Jean-Rémi King. Language processing in brains and deep neural networks: computational convergence and its limits. *BioRxiv*, 2020.

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single! vector: Probing sentence embeddings for linguistic properties. In *ACL 2018-56th Annual Meeting of the Association for Computational Linguistics*, volume 1, pp. 2126–2136. Association for Computational Linguistics, 2018.

Rutvik Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system, 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*, 2018.

E. Fedorenko, P.-J. Hsieh, A. Nieto-Castanon, S. Whitfield-Gabrieli, and N. Kanwisher. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. *Journal of Neurophysiology*, 104(2):1177–1194, 2010.

Evelina Fedorenko and Sharon L Thompson-Schill. Reworking the language network. *Trends in cognitive sciences*, 18(3):120–126, 2014.

Angela D Friederici. The cortical language circuit: from auditory perception to sentence comprehension. *Trends in cognitive sciences*, 16(5):262–268, 2012.

Angela D Friederici, Shirley-Ann Rüschemeyer, Anja Hahne, and Christian J Fiebach. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. *Cerebral cortex*, 13(2):170–177, 2003.

Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, 2019.

Adam Gazzaley and Anna C Nobre. Top-down modulation: bridging selective attention and working memory. *Trends in cognitive sciences*, 16(2):129–135, 2012.

Christopher R Genovese. A bayesian time-course model for functional magnetic resonance imaging data. *Journal of the American Statistical Association*, 95(451):691–703, 2000.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. *Nature*, 536(7615):171–178, 2016.

Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. *Nature neuroscience*, 25(3):369–380, 2022.

Olaf Hauk and Friedemann Pulvermüller. Effects of word length and frequency on the human event-related potential. *Clinical Neurophysiology*, 115(5):1090–1103, 2004.

Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang. Cognival: A framework for cognitive word embedding evaluation. In *CoNLL*, pp. 538–549, 2019.

Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. *Journal of Artificial Intelligence Research*, 61:907–926, 2018.

Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fMRI. In *Advances in Neural Information Processing Systems*, pp. 6628–6637, 2018.

S Jat, H Tang, P Talukdar, and T Mitchel. Relating simple sentence representations in deep neural networks and the brain. In *ACL*, pp. 5137–5154, 2020.

Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In *ACL 2019-57th Annual Meeting of the Association for Computational Linguistics*, 2019.

Sreejan Kumar, Theodore R Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. *bioRxiv*, 2022.

Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In *Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations*, pp. 55–60, 2014.

Michael McCloskey. Networks and theories: The place of connectionism in cognitive science. *Psychological science*, 2(6):387–395, 1991.

Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word-level semantics and prediction. *arXiv preprint arXiv:2212.00596*, 2022.

Camille K Milton, Vukshitha Dhanaraj, Isabella M Young, Hugh M Taylor, Peter J Nicholas, Robert G Briggs, Michael Y Bai, Rannulu D Fonseka, Jorge Hormovas, Yueh-Hsin Lin, et al. Parcellation-based anatomic model of the semantic network. *Brain and behavior*, 11(4):e02065, 2021.

Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. Exploring the role of bert token representations to explain sentence probing results. In *The 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 792–806. Association for Computational Linguistics, 2021.

Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. *Scientific data*, 8(1): 1–22, 2021.

Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. *Current biology*, 21(19):1641–1646, 2011.

Subba Reddy Oota, Vijay Rowtula, Manish Gupta, and Raju S Bapi. Stepencog: A convolutional lstm autoencoder for near-perfect fmri encoding. In *IJCNN*, pp. 1–8. IEEE, 2019.
Subba Reddy Oota, Frederic Alexandre, and Xavier Hinaut. Long-term plausibility of language models and neural dynamics during narrative listening. In *Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 44, 2022a.

Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? *NAACL*, 2022b.

Subba Reddy Oota, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampud. Syntactic structure processing in the brain while listening. *Findings of the Association for Computational Linguistics: ACL*, 2023.

Christophe Pallier, Anne-Dominique Devauchelle, and Stanislas Dehaene. Cortical representation of the constituent structure of sentences. *Proceedings of the National Academy of Sciences*, 108(6): 2522-2527, 2011.

Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Decoding of generic mental representations from functional mri data using word embeddings. *bioRxiv*, pp. 057216, 2016.

Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. *Nature communications*, 9(1):1-13, 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. *OpenAI blog*, 1(8):9, 2019.

Gregor Rainer, Wael F Asaad, and Earl K Miller. Memory fields of neurons in the primate prefrontal cortex. *Proceedings of the National Academy of Sciences*, 95(25):15008-15013, 1998.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. *Association for Computational Linguistics*, 2020.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. *Transactions of the Association for Computational Linguistics*, 8:842-866, 2020.

Edmund T Rolls, Gustavo Deco, Chu-Chung Huang, and Jianfeng Feng. The human language effective connectome. *NeuroImage*, pp. 119352, 2022.

Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. *Proceedings of the National Academy of Sciences*, 118(45), 2021.

Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. *Advances in neural information processing systems*, 32, 2019.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Towards sentence-level brain decoding with distributed representations. In *AAAI*, pp. 7047-7054, 2019.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Neural encoding and decoding with distributed sentence representations. *IEEE Transactions on Neural Networks and Learning Systems*, 32(2):589-603, 2020.

Andreï Nikolaevich Tikhonov, Vasilij Ja Arsenin, and Vasiliï Arsenin. *Solutions of ill-posed problems*. V H Winston, 1977.

Mariya Toneva. *Bridging Language in Machines with Language in the Brain*. PhD thesis, Carnegie Mellon University, 2021.

Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). *Advances in Neural Information Processing Systems*, 32, 2019.
Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. *Nature Computational Science*, 2(11):745-757, 2022.

Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. Fine-grained neural decoding with distributed word representations. *Information Sciences*, 507:256-272, 2020.

Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical models of language with brain activity during reading. In *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing*, pp. 233–243, 2014.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art
natural language processing. In *Proceedings of the 2020 conference on empirical methods in
natural language processing: system demonstrations*, pp. 38–45, 2020.

Theodore P Zanto, Michael T Rubens, Arul Thangavel, and Adam Gazzaley. Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory. *Nature neuroscience*, 14(5):656–661, 2011.
# Appendix

## A Distribution of Class Labels Across Each Probing Task

Figure 5 reports the distribution of class labels across each linguistic property. For the probing tasks such as Sentence Length, Tree Depth, and Top Constituents, we balance the number of classes to overcome the imbalance problem. We balance the classes for these three probing tasks as follows: (i) Sentence Length: 3-classes ($\le5$, 5-8 and $\ge9$), (ii) TreeDepth: 3-classes (5, 6-7 and $\ge8$), and TopConstituents: 2-classes (1, $\ge2$).

<figure><img src="image_7.png" /><figcaption>Figure 5: Distribution of class labels across each linguistic property. For the probing tasks such as Sentence Length, Tree Depth, and Top Constituents, we balance the number of classes to overcome the imbalance problem.</figcaption></figure>

Figure 6 displays the common samples between class labels of pair of probing tasks. This reports whether the cells are balanced across class labels of different probing tasks.
<figure><img src="image_8.png" /><figcaption>Figure 6: Number of common samples between pair of class labels across probing tasks.</figcaption></figure>

## B INLP Projection vs. Our Method

We also implemented the Iterative Null-Space Projection (INLP) method (Ravfogel et al., 2020) to verify whether our removal method performance is similar to previously proposed method. We found the results to be similar. Results using our method are in Table 4. Results using the INLP method are below.
<table><caption>Table 4: Word-Level: INLP Projection Method vs Our Removal Method: Probing task accuracy for each BERT layer after removal of each linguistic property using the 21st year stimuli.</caption><thead><tr><th>Layers</th><th>Sentence Length</th><th>TreeDepth</th><th>TopConstituents</th><th>Tense</th><th>Subject Number</th><th>Object Number</th></tr></thead><tbody><tr><td>1</td><td>38.23</td><td>57.92</td><td>49.36</td><td>50.75</td><td>50.50</td><td>50.89</td></tr><tr><td>2</td><td>37.03</td><td>34.36</td><td>41.47</td><td>44.13</td><td>47.61</td><td>49.20</td></tr><tr><td>3</td><td>37.20</td><td>28.94</td><td>36.76</td><td>59.12</td><td>48.37</td><td>47.73</td></tr><tr><td>4</td><td>38.71</td><td>28.54</td><td>54.72</td><td>40.81</td><td>48.04</td><td>56.03</td></tr><tr><td>5</td><td>39.18</td><td>41.17</td><td>47.27</td><td>41.54</td><td>60.27</td><td>54.48</td></tr><tr><td>6</td><td>37.96</td><td>33.19</td><td>46.97</td><td>34.82</td><td>48.24</td><td>50.88</td></tr><tr><td>7</td><td>39.43</td><td>35.97</td><td>36.94</td><td>45.68</td><td>50.09</td><td>60.27</td></tr><tr><td>8</td><td>38.85</td><td>27.32</td><td>36.94</td><td>48.77</td><td>58.09</td><td>47.18</td></tr><tr><td>9</td><td>37.00</td><td>27.99</td><td>49.46</td><td>64.99</td><td>60.50</td><td>48.65</td></tr><tr><td>10</td><td>38.87</td><td>36.69</td><td>36.94</td><td>45.28</td><td>56.16</td><td>47.75</td></tr><tr><td>11</td><td>36.15</td><td>42.68</td><td>51.93</td><td>36.27</td><td>57.65</td><td>62.87</td></tr><tr><td>12</td><td>38.14</td><td>51.33</td><td>55.19</td><td>31.19</td><td>49.81</td><td>47.04</td></tr><tr><td>Avg (INLP)</td><td>38.06</td><td>37.17</td><td>45.32</td><td>45.27</td><td>52.94</td><td>51.91</td></tr><tr><td>Avg (Ours)</td><td>45.30</td><td>41.77</td><td>46.30</td><td>46.36</td><td>51.75</td><td>50.24</td></tr><tr><td>Chance (Probability)</td><td>42.76</td><td>50.39</td><td>53.63</td><td>66.31</td><td>83.24</td><td>80.78</td></tr></tbody></table>

## B.1 Probing Analysis While Removing Information about a Property and Testing Another

We run the probing analysis while removing information about a property and testing another, to see if only information specific about a task is being removed at a time. The detailed result of each probing task is presented Tables 5, 6, 7, 8, 9, and 10. We observe that removal of a task (property) does not affect the performance for other tasks except for TreeDepth and TopConstituents.

<table><caption>Table 5: Layer-wise Probing task performance for Sentence Length task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR Sentence Length</th><th>AR Sentence Length</th><th>AR TreeDepth</th><th>AR TopConstituents</th><th>AR Tense</th><th>AR SubjNum</th><th>AR ObjNum</th></tr></thead><tbody><tr><td>1</td><td><strong>74.67</strong></td><td>43.28</td><td>69.47</td><td>72.37</td><td>72.24</td><td>71.52</td><td>73.58</td></tr><tr><td>2</td><td>69.83</td><td>42.44</td><td>70.37</td><td>73.09</td><td>73.28</td><td>73.04</td><td>74.18</td></tr><tr><td>3</td><td>72.31</td><td>46.19</td><td>71.22</td><td>72.85</td><td>73.94</td><td>73.22</td><td>74.12</td></tr><tr><td>4</td><td>71.34</td><td>46.43</td><td>71.64</td><td>71.52</td><td>73.40</td><td>74.00</td><td>73.52</td></tr><tr><td>5</td><td>72.67</td><td>46.97</td><td>71.10</td><td>74.37</td><td>76.24</td><td>74.06</td><td>76.54</td></tr><tr><td>6</td><td>70.38</td><td>44.37</td><td>69.89</td><td>72.91</td><td>74.61</td><td>74.00</td><td>74.79</td></tr><tr><td>7</td><td>72.98</td><td>46.55</td><td>71.22</td><td>71.52</td><td>75.03</td><td>74.12</td><td>74.30</td></tr><tr><td>8</td><td>72.67</td><td>44.67</td><td>70.07</td><td>70.98</td><td>74.06</td><td>74.73</td><td>72.73</td></tr><tr><td>9</td><td>70.50</td><td>45.28</td><td>70.07</td><td>69.35</td><td>72.31</td><td>73.16</td><td>71.83</td></tr><tr><td>10</td><td>72.91</td><td>47.93</td><td>68.92</td><td>70.92</td><td>71.89</td><td>71.95</td><td>72.49</td></tr><tr><td>11</td><td>70.07</td><td>46.67</td><td>67.35</td><td>70.49</td><td>73.06</td><td>71.64</td><td>71.83</td></tr><tr><td>12</td><td>71.77</td><td>42.93</td><td>66.92</td><td>68.62</td><td>70.31</td><td>71.64</td><td>71.40</td></tr><tr><td>Avg</td><td>71.84</td><td>45.30</td><td>64.01</td><td>71.58</td><td>73.36</td><td>73.09</td><td>73.44</td></tr></tbody></table>

<table><caption>Table 6: Layer-wise Probing task performance for TreeDepth task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR TreeDepth</th><th>AR TreeDepth</th><th>AR SentenceLength</th><th>AR TopConstituents</th><th>AR Tense</th><th>AR SubjectNumber</th><th>AR ObjectNumber</th></tr></thead><tbody><tr><td>1</td><td>76.30</td><td>42.93</td><td>76.84</td><td>77.62</td><td>77.69</td><td>75.93</td><td>76.78</td></tr><tr><td>2</td><td>76.72</td><td>38.88</td><td>76.48</td><td>76.96</td><td>76.90</td><td>77.63</td><td>76.54</td></tr><tr><td>3</td><td>75.76</td><td>40.33</td><td>77.97</td><td>76.42</td><td>79.62</td><td>76.84</td><td>75.57</td></tr><tr><td>4</td><td>75.94</td><td>38.63</td><td>77.20</td><td>77.08</td><td>78.42</td><td>77.38</td><td>75.82</td></tr><tr><td>5</td><td>76.00</td><td>40.88</td><td>76.48</td><td>76.36</td><td>78.11</td><td>77.67</td><td>76.23</td></tr><tr><td>6</td><td><strong>79.02</strong></td><td>41.89</td><td>76.17</td><td>76.60</td><td>78.11</td><td>76.30</td><td>77.81</td></tr><tr><td>7</td><td>77.93</td><td>41.23</td><td>77.14</td><td>77.56</td><td>77.50</td><td>77.50</td><td>77.81</td></tr><tr><td>8</td><td>76.07</td><td>40.08</td><td>76.05</td><td>76.66</td><td>76.36</td><td>76.06</td><td>76.84</td></tr><tr><td>9</td><td>77.15</td><td>42.62</td><td>76.90</td><td>77.08</td><td>77.93</td><td>78.47</td><td>77.38</td></tr><tr><td>10</td><td>76.90</td><td>41.78</td><td>76.05</td><td>76.54</td><td>74.84</td><td>76.29</td><td>76.42</td></tr><tr><td>11</td><td>77.27</td><td>45.47</td><td>75.15</td><td>76.42</td><td>76.17</td><td>77.56</td><td>77.14</td></tr><tr><td>12</td><td>76.39</td><td>46.61</td><td>75.99</td><td>76.23</td><td>77.75</td><td>75.09</td><td>76.01</td></tr><tr><td>Average</td><td>76.79</td><td>41.77</td><td>76.56</td><td>76.70</td><td>77.75</td><td>75.09</td><td>76.70</td></tr></tbody></table>

## C Layer-wise Whole Brain Analysis before/after removal of linguistic properties.

In Figure 7, we report the layer-wise performance for pretrained BERT before and after the removal of each of the linguistic properties. Similar to previous work (Toneva & Wehbe, 2019), we observe that pretrained BERT has best brain alignment in the middle layers across all properties. We further observe that the alignment after removing the linguistic property is significantly worse mainly for middle to late layers. Note that the layers at which there is a significant difference are indicated with a red dot. This pattern holds across all of the linguistic properties that we tested. These results provide
<table><caption>Table 7: Layer-wise Probing task performance for TopConstituents task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR TopConstituents</th><th>AR TopConstituents</th><th>AR SentenceLength</th><th>AR TreeDepth</th><th>AR Tense</th><th>AR SubjectNumber</th><th>AR ObjectNumber</th></tr></thead><tbody><tr><td>1</td><td>77.15</td><td>47.28</td><td>78.17</td><td>78.11</td><td>76.72</td><td>77.39</td><td>77.81</td></tr><tr><td>2</td><td>78.60</td><td>42.75</td><td>77.69</td><td>78.54</td><td>78.41</td><td>77.81</td><td>77.93</td></tr><tr><td>3</td><td>77.81</td><td>48.85</td><td>76.36</td><td>77.15</td><td>77.15</td><td>77.69</td><td>77.75</td></tr><tr><td>4</td><td>78.36</td><td>48.00</td><td>77.69</td><td>77.03</td><td>77.81</td><td>78.23</td><td>77.69</td></tr><tr><td>5</td><td>78.60</td><td>45.28</td><td>77.87</td><td>78.42</td><td>78.35</td><td>77.93</td><td>78.36</td></tr><tr><td>6</td><td><strong>80.23</strong></td><td>43.47</td><td>78.71</td><td>79.20</td><td>77.21</td><td>78.54</td><td>78.65</td></tr><tr><td>7</td><td><strong>80.23</strong></td><td>46.43</td><td>78.65</td><td>78.66</td><td>77.39</td><td>79.02</td><td>79.38</td></tr><tr><td>8</td><td>78.90</td><td>46.86</td><td>77.63</td><td>78.36</td><td>78.41</td><td>79.08</td><td>79.44</td></tr><tr><td>9</td><td>79.87</td><td>44.55</td><td>77.76</td><td>78.30</td><td>78.71</td><td>79.87</td><td>78.89</td></tr><tr><td>10</td><td>78.17</td><td>47.76</td><td>77.81</td><td>77.87</td><td>78.41</td><td>78.96</td><td>78.42</td></tr><tr><td>11</td><td>77.69</td><td>45.77</td><td>78.42</td><td>78.84</td><td>77.03</td><td>78.00</td><td>78.00</td></tr><tr><td>12</td><td>78.29</td><td>48.67</td><td>76.36</td><td>78.78</td><td>75.21</td><td>78.30</td><td>78.36</td></tr><tr><td>Average</td><td>78.66</td><td>46.30</td><td>77.76</td><td>78.27</td><td>78.16</td><td>78.40</td><td>78.39</td></tr></tbody></table>

<table><caption>Table 8: Layer-wise Probing task performance for Tense task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR Tense</th><th>AR Tense</th><th>AR SentenceLength</th><th>AR TreeDepth</th><th>AR TopConstituents</th><th>AR SubjectNumber</th><th>AR ObjectNumber</th></tr></thead><tbody><tr><td>1</td><td>87.00</td><td>59.25</td><td>87.00</td><td>86.22</td><td>85.91</td><td>85.07</td><td>86.82</td></tr><tr><td>2</td><td>87.18</td><td>48.25</td><td>88.15</td><td>87.06</td><td>87.54</td><td>86.22</td><td>86.64</td></tr><tr><td>3</td><td>87.42</td><td>44.26</td><td>87.55</td><td>87.30</td><td>87.48</td><td>87.55</td><td>87.18</td></tr><tr><td>4</td><td>88.09</td><td>42.56</td><td>87.42</td><td>87.49</td><td>86.82</td><td>87.36</td><td>87.73</td></tr><tr><td>5</td><td>88.39</td><td>44.26</td><td>88.03</td><td>88.08</td><td>87.73</td><td>87.12</td><td>87.67</td></tr><tr><td>6</td><td>87.17</td><td>44.44</td><td>86.33</td><td>86.70</td><td>86.22</td><td>87.42</td><td>86.46</td></tr><tr><td>7</td><td>88.69</td><td>42.62</td><td>86.52</td><td>88.33</td><td>88.09</td><td>87.30</td><td>87.79</td></tr><tr><td>8</td><td>87.42</td><td>44.56</td><td>86.82</td><td>87.24</td><td>86.64</td><td>86.52</td><td>84.95</td></tr><tr><td>9</td><td>88.27</td><td>47.22</td><td>87.79</td><td>87.84</td><td>87.73</td><td>87.67</td><td>87.48</td></tr><tr><td>10</td><td><strong>88.94</strong></td><td>45.47</td><td>88.75</td><td>88.21</td><td>86.15</td><td>87.90</td><td>88.21</td></tr><tr><td>11</td><td>87.24</td><td>48.43</td><td>86.88</td><td>87.42</td><td>86.94</td><td>87.00</td><td>86.46</td></tr><tr><td>12</td><td>86.88</td><td>45.10</td><td>86.40</td><td>86.64</td><td>85.31</td><td>85.97</td><td>86.40</td></tr><tr><td>Average</td><td>87.72</td><td>46.36</td><td>87.30</td><td>87.38</td><td>86.88</td><td>86.93</td><td>86.98</td></tr></tbody></table>

<table><caption>Table 9: Layer-wise Probing task performance for Subject Number task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR SubjectNumber</th><th>AR SubjectNumber</th><th>AR SentenceLength</th><th>AR TreeDepth</th><th>AR TopConstituents</th><th>AR Tense</th><th>AR ObjectNumber</th></tr></thead><tbody><tr><td>1</td><td>87.00</td><td>59.25</td><td>87.57</td><td>87.76</td><td>87.88</td><td>87.56</td><td>71.61</td></tr><tr><td>2</td><td>92.32</td><td>55.50</td><td>86.40</td><td>86.57</td><td>86.45</td><td>86.15</td><td>69.82</td></tr><tr><td>3</td><td>93.04</td><td>48.55</td><td>86.84</td><td>86.94</td><td>86.77</td><td>86.91</td><td>70.17</td></tr><tr><td>4</td><td>93.50</td><td>50.12</td><td>86.56</td><td>86.52</td><td>86.68</td><td>86.97</td><td>70.84</td></tr><tr><td>5</td><td>94.05</td><td>49.88</td><td>87.78</td><td>87.87</td><td>87.59</td><td>87.70</td><td>70.07</td></tr><tr><td>6</td><td>94.98</td><td>55.08</td><td>87.59</td><td>87.77</td><td>87.76</td><td>87.88</td><td>70.96</td></tr><tr><td>7</td><td>95.88</td><td>50.24</td><td>87.84</td><td>87.94</td><td>87.56</td><td>87.76</td><td>70.66</td></tr><tr><td>8</td><td>96.10</td><td>50.24</td><td>87.35</td><td>87.65</td><td>87.04</td><td>86.85</td><td>71.13</td></tr><tr><td>9</td><td>96.38</td><td>52.78</td><td>87.52</td><td>87.55</td><td>87.32</td><td>87.09</td><td>70.50</td></tr><tr><td>10</td><td>96.06</td><td>53.68</td><td>87.78</td><td>8.07</td><td>87.80</td><td>87.31</td><td>70.91</td></tr><tr><td>11</td><td><strong>96.94</strong></td><td>53.44</td><td>87.29</td><td>87.49</td><td>87.47</td><td>86.97</td><td>70.81</td></tr><tr><td>12</td><td>94.03</td><td>51.45</td><td>87.44</td><td>87.55</td><td>87.64</td><td>87.23</td><td>70.84</td></tr><tr><td>Average</td><td>94.19</td><td>51.75</td><td>87.33</td><td>87.46</td><td>87.33</td><td>87.20</td><td>70.69</td></tr></tbody></table>

<table><caption>Table 10: Layer-wise Probing task performance for Object Number task. Note in tables below AR=After Removal and BR=before removal.</caption><thead><tr><th>Layers</th><th>BR ObjectNumber</th><th>AR ObjectNumber</th><th>AR SentenceLength</th><th>AR TreeDepth</th><th>AR TopConstituents</th><th>AR Tense</th><th>AR SubjectNumber</th></tr></thead><tbody><tr><td>1</td><td>93.28</td><td>47.31</td><td>86.63</td><td>86.68</td><td>86.77</td><td>86.83</td><td>71.45</td></tr><tr><td>2</td><td>93.47</td><td>54.59</td><td>86.55</td><td>86.61</td><td>86.63</td><td>86.55</td><td>70.20</td></tr><tr><td>3</td><td>93.80</td><td>49.76</td><td>86.88</td><td>86.84</td><td>86.85</td><td>86.62</td><td>70.15</td></tr><tr><td>4</td><td>94.90</td><td>50.06</td><td>85.81</td><td>85.87</td><td>85.87</td><td>85.79</td><td>71.14</td></tr><tr><td>5</td><td>93.59</td><td>51.45</td><td>86.17</td><td>86.42</td><td>86.48</td><td>86.48</td><td>70.05</td></tr><tr><td>6</td><td>94.50</td><td>54.17</td><td>86.27</td><td>86.45</td><td>86.19</td><td>86.74</td><td>70.61</td></tr><tr><td>7</td><td>94.62</td><td>47.58</td><td>86.38</td><td>86.42</td><td>86.33</td><td>86.39</td><td>71.00</td></tr><tr><td>8</td><td><strong>95.10</strong></td><td>50.18</td><td>85.86</td><td>85.79</td><td>85.79</td><td>85.57</td><td>70.76</td></tr><tr><td>9</td><td>94.56</td><td>49.27</td><td>85.63</td><td>85.61</td><td>85.55</td><td>85.46</td><td>70.20</td></tr><tr><td>10</td><td>94.50</td><td>50.30</td><td>86.21</td><td>86.16</td><td>86.21</td><td>86.16</td><td>70.73</td></tr><tr><td>11</td><td>94.92</td><td>49.52</td><td>84.94</td><td>84.95</td><td>84.90</td><td>85.01</td><td>70.33</td></tr><tr><td>12</td><td>93.95</td><td>48.73</td><td>85.49</td><td>85.33</td><td>85.65</td><td>85.30</td><td>70.49</td></tr><tr><td>Average</td><td>94.27</td><td>50.24</td><td>86.07</td><td>86.11</td><td>86.10</td><td>86.09</td><td>70.59</td></tr></tbody></table>

direct evidence that these linguistic properties in fact significantly Figure 7 affect the alignment between fMRI recordings and pretrained BERT.
## D Layer-wise Language ROIs Analysis before/after removal of linguistic properties.

Here, we examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprehension (Fedorenko et al., 2010; Fedorenko & Thompson-Schill, 2014) and word semantics (Binder et al., 2009). We find that across language regions, the alignment is significantly decreased by the removal of the linguistic properties across all layers, as shown in Figure 8.

## E Layer-wise Language sub-ROIs Analysis before/after removal of linguistic properties.

We further investigate the language sub-regions, such as 44, 45, IFJa, IFSp, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Figure 9).

Each language brain region is not necessarily homogeneous in function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. We thus further analyze several important language sub-regions that are thought to exemplify the variety of functionality across the broader language regions (Rolls et al., 2022): STGa, STSda, 44, 45, 55b, STV, SFL, PFm, PGi, PGs, IFJa and IFSp in the main paper. Here we extend this analysis to more sub-regions: PSL, STV, SFL, PFm, PGs, PGi. We demonstrate the correlations for each language brain sub-region separately in Table 3 in the main paper.

<figure><img src="image_9.png" /><figcaption>Figure 7: Balanced Classes: Model trained on pretrained BERT features and removal of different linguistic properties. Each plot compares the layer-wise performance of pretrained BERT and removal of each probing task. Bottom plot displays the pretrained BERT vs. removal of all tasks. A red * at a particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 8: Avg Pearson correlations for language regions AG, ATL, PTL, IFG, MFG, IFGOrb, PCC and dmPFC. Model trained on pretrained BERT features and removal of different linguistic properties (shown only for important properties for clarity). Each plot compares the layer-wise performance of pretrained BERT and removal of each probing task.</figcaption></figure>

## F Probing Results: GPT2

Like the probing experiments with BERT in the main paper, we also perform experiments with GPT2. We find the results to be similar to BERT, i.e., a rich hierarchy of linguistic signals: initial to middle layers encode surface information, middle layers encode syntax, middle to top layers encode semantics. Table 11 reports the result for each probing task, before and after removal of the linguistic property from pretrained GPT2. We verify that the removal of each linguistic property from GPT2 leads to reduced task performance across all layers, as expected. We also report the layer-wise performance for pretrained GPT2 before and after the removal of one representative linguistic property (TopConstituents) in Fig. 11. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure).
<figure><img src="image_11.png" /><figcaption>Figure 9: We extend analyses in Fig. 7 to some language sub-regions 45, IFJa, IFSp, 55b, STGa, A5, PSL, TPOJ1, STSda, STSdp and STV. Please refer to caption of Fig. 7 for detailed understanding of each plot.</figcaption></figure>

<table><caption>Table 11: Word-Level Probing task performance for each GPT2 layer before and after removal of each linguistic property using the 21st year stimuli.</caption><thead><tr><th rowspan="2">Layers</th><th colspan="2">Sentence Length<br>3-classes<br>(Surface)</th><th colspan="2">TreeDepth<br>3-classes<br>(Syntactic)</th><th colspan="2">TopConstituents<br>2-classes<br>(Syntactic)</th><th colspan="2">Tense<br>2-classes<br>(Semantic)</th><th colspan="2">Subject Number<br>2-classes<br>(Semantic)</th><th colspan="2">Object Number<br>2-classes<br>(Semantic)</th></tr><tr><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th><th>before</th><th>after</th></tr></thead><tbody><tr><td>1</td><td><strong>54.11</strong></td><td>30.48</td><td>65.78</td><td>32.78</td><td>60.88</td><td>45.56</td><td>76.59</td><td>49.78</td><td>83.57</td><td>49.90</td><td>87.17</td><td>50.89</td></tr><tr><td>2</td><td>54.00</td><td>32.50</td><td>66.62</td><td>33.65</td><td>60.52</td><td>44.55</td><td>77.57</td><td>50.79</td><td>87.87</td><td>50.91</td><td>87.60</td><td>50.88</td></tr><tr><td>3</td><td>53.02</td><td>32.51</td><td>66.26</td><td>33.67</td><td>60.46</td><td>49.63</td><td>77.97</td><td>50.78</td><td>87.94</td><td>50.91</td><td>87.97</td><td>50.91</td></tr><tr><td>4</td><td>52.90</td><td>33.54</td><td>67.38</td><td>34.64</td><td>60.94</td><td>49.63</td><td>77.98</td><td>50.78</td><td>88.84</td><td>50.51</td><td>88.10</td><td>52.12</td></tr><tr><td>5</td><td>52.94</td><td>32.53</td><td>67.78</td><td>33.46</td><td>61.43</td><td>42.34</td><td>77.85</td><td>50.39</td><td>89.03</td><td>51.05</td><td>88.42</td><td>51.87</td></tr><tr><td>6</td><td>52.47</td><td>33.55</td><td>68.14</td><td>33.23</td><td><strong>62.21</strong></td><td>48.62</td><td>78.17</td><td>50.79</td><td>89.50</td><td>50.90</td><td>88.81</td><td>50.31</td></tr><tr><td>7</td><td>52.24</td><td>33.53</td><td><strong>68.68</strong></td><td>34.55</td><td>61.43</td><td>48.38</td><td>78.02</td><td>51.68</td><td>89.89</td><td>50.95</td><td>88.98</td><td>50.88</td></tr><tr><td>8</td><td>52.00</td><td>33.52</td><td>68.08</td><td>31.22</td><td>61.43</td><td>49.39</td><td>78.21</td><td>51.81</td><td>89.81</td><td>51.81</td><td>89.10</td><td>52.12</td></tr><tr><td>9</td><td>53.03</td><td>32.53</td><td>67.47</td><td>33.01</td><td>61.19</td><td>49.62</td><td><strong>78.22</strong></td><td>51.17</td><td>90.23</td><td>50.91</td><td><strong>89.14</strong></td><td>52.75</td></tr><tr><td>10</td><td>52.47</td><td>32.52</td><td>67.53</td><td>33.67</td><td>61.37</td><td>48.61</td><td>78.20</td><td>49.57</td><td>90.17</td><td>49.39</td><td>89.09</td><td>51.73</td></tr><tr><td>11</td><td>52.53</td><td>32.50</td><td>67.47</td><td>34.64</td><td>61.19</td><td>48.63</td><td>77.96</td><td>50.78</td><td><strong>90.56</strong></td><td>51.63</td><td>89.08</td><td>52.53</td></tr><tr><td>12</td><td>52.83</td><td>33.53</td><td>66.44</td><td>34.27</td><td>61.06</td><td>48.61</td><td>77.62</td><td>49.78</td><td>90.10</td><td>51.07</td><td>89.06</td><td>54.34</td></tr></tbody></table>
<figure><img src="image_12.png" /><figcaption>Figure 10: For whole brain, paired t-test was performed to identify the layers and removal of linguistic properties where the pretrained BERT model has greater brain alignment than the removal of each linguistic property, with statistical significance. p-values obtained from paired t-test, after false discovery rate (FDR) correction using the Benjamini-Hochberg (BH) procedure.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 11: Comparison of layer-wise performance of pretrained GPT2 and removal of one linguistic property—TopConstituents.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 12: Voxel-wise correlations across layers between brain alignment of pretrained BERT before and after removing all linguistic properties.</figcaption></figure>