Title: Joint processing of linguistic properties in brains and language models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Linguistic properties

fMRI

probing tasks

cognitive neuroscience

language models

NLP
Abstract: Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available https://github.com/subbareddy248/lingprop-brain-alignment.

Joint processing of linguistic properties in brains and language models

Abstract

Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available 1 .

1 Introduction

Language models that have been pretrained for the next word prediction task using millions of text documents can significantly predict brain recordings of people comprehending language (Wehbe et al., 2014; Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King, 2020; Schrimpf et al., 2021; Goldstein et al., 2022). Understanding the reasons behind the observed similarities between language comprehension in machines and brains can lead to more insight into both systems.

While such similarities have been observed at a coarse level, it is not yet clear whether and how the two systems align in their information processing pipeline. This pipeline has been studied separately in both systems. In natural language processing (NLP), researchers use probing tasks to uncover the parts of the model that encode specific linguistic properties (e.g. sentence length, tree depth, top constituents, tense, bigram shift, subject number, object number) (Adi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018; Jawahar et al., 2019; Rogers et al., 2020). These techniques have revealed a hierarchy of information processing in multi-layered language models that progresses from simple to complex with increased depth. In cognitive neuroscience, traditional experiments study the processing of specific linguistic properties by carefully controlling the experimental stimulus and observing the locations or time points of processing in the brain that are affected the most by the controlled stimulus (Hauk & Pulvermüller, 2004; Pallier et al., 2011).

More recently, researchers have begun to study the alignment of these brain language regions with the layers of language models, and found that the best alignment was achieved in the middle layers of these models (Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King, 2020). This has been

fMRI

Linguistic

property

Language

model

Significant

difference

Ling.

prop. affects

alignment

Residual =

Original encoding

performance

Residual encoding

performance

Naturalistic

stimulus

This is Los Angeles.

And it's the …

hypothesized to be because the middle layers may contain the most high-level language information as they are farthest from the input and output layers, which contain word-level information due to the self-supervised training objective. However, this hypothesis is difficult to reconcile with the results from more recent NLP probing tasks Conneau et al. (2018); Jawahar et al. (2019), which suggest that the deepest layers in the model should represent the highest-level language information. Taken together, these findings open up the question of what linguistic information underlies the observed alignment between brains and language models.

Our work aims to examine this question via a direct approach (see Figure 1 for a schematic). For a number of linguistic properties, we analyze how the alignment between brain recordings and language model representations is affected by the elimination of information related to each linguistic property. For the purposes of this work, we focus on one popular language model-BERT (Devlin et al., 2018)-which has both been studied extensively in the NLP interpretability literature (i.e. BERTology Jawahar et al. (2019)) and has been previously shown to significantly predict fMRI recordings of people processing language (Toneva & Wehbe, 2019; Schrimpf et al., 2021). We test the effect of a range of linguistic properties that have been previously shown to be represented in pretrained BERT (Jawahar et al., 2019). We use a dataset of fMRI recordings that are openly available (Nastase et al., 2021) and correspond to 18 participants listening to a natural story.

Using this direct approach, we find that the elimination of each linguistic property results in a significant decrease in brain alignment across all layers of BERT. We additionally find that the syntactic properties (Top Constituents and Tree Depth) have the highest effect on the trend of brain alignment across model layers. Specifically, Top Constituents is responsible for the bump in brain alignment in the middle layers for all language regions whereas Tree Depth has an impact for temporal (ATL and PTL) and frontal language regions (IFG and MFG). Performing the same analyses with a second popular language model, GPT2 (Radford et al., 2019), yielded similar results (see Appendix section F).

Our main contributions are as follows:

We propose a direct approach to evaluate the joint processing of linguistic properties in brains and language models.

We show that removing specific linguistic properties leads to a significant decrease in brain alignment. We find that the tested syntactic properties are the most responsible for the trend of brain alignment across BERT layers.

Detailed region and sub-region analysis reveal that properties that may not impact the whole brain alignment trend, may play a significant role in local trends (e.g., Object Number for

ATL and IFGOrb regions, Tense for PCC regions, Sentence Length and Subject Number for PFm sub-region).

We make the code publicly available 1 .

2 Related Work

Our work is most closely related to that of Toneva et al. (2022), who employ a similar residual approach to study the supra-word meaning of language by removing the contribution of individual words to brain alignment. We build on this approach to study the effect of specific linguistic properties on brain alignment across layers of a language model. Other recent work has examined individual attention heads in BERT and shown that their performance on syntactic tasks correlate with their brain encoding performance in several brain regions (Kumar et al., 2022). Our work presents a complementary approach that gives direct evidence for the importance of a linguistic property to brain alignment.

This work also relates to previous works that investigated the linguistic properties encoded across the layer hierarchy of language models (Adi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018; Jawahar et al., 2019; Rogers et al., 2020). Unlike these studies in NLP, we focus on understanding the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.

Our work also relates to a growing literature that relates representations of words in language models to those in the brain. A number of studies have related brain responses to word embedding methods (Pereira et al., 2016; Anderson et al., 2017; Pereira et al., 2018; Toneva & Wehbe, 2019; Hollenstein et al., 2019; Wang et al., 2020), sentence representation models (Sun et al., 2019; Toneva &Wehbe, 2019; Sun et al., 2020), recurrent neural networks (Jain & Huth, 2018; Oota et al., 2019), and Transformer methods (Gauthier & Levy, 2019; Toneva & Wehbe, 2019; Schwartz et al., 2019; Jat et al., 2020; Schrimpf et al., 2021; Goldstein et al., 2022; Oota et al., 2022b,a; Merlin & Toneva, 2022; Aw & Toneva, 2023; Oota et al., 2023). Our approach is complementary to these previous works and can be used to further understand the reasons behind the observed brain alignment.

3 Dataset Curation

Brain Imaging Dataset The 'Narratives' collection aggregates a variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. We analyze the 'Narratives-21 st year' dataset (Nastase et al., 2021), which is one of the largest publicly available fMRI datasets (in terms of number of samples per participant). The dataset contains data from 18 subjects who listened to the story titled '21 st year'. The dataset for each subject contains 2226 samples (TRs-Time Repetition). The dataset was already preprocessed and projected on the surface space ('fsaverage6'). We use the multi-modal parcellation of the human cerebral cortex (Glasser Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps (Glasser et al., 2016), since the Narratives dataset contains annotations that correspond to this atlas. The data covers seven language brain regions of interest (ROIs) in the human brain with the following subdivisions: (i) angular gyrus (AG: PFm, PGs, PGi, TPOJ2, and TPOJ3); (ii) anterior temporal lobe (ATL: STSda, STSva, STGa, TE1a, TE2a, TGv, and TGd); (iii) posterior temporal lobe (PTL: A4, A5, STSdp, STSvp, PSL, STV, TPOJ1); (iv) inferior frontal gyrus (IFG: 44, 45, IFJa, IFSp); (v) middle frontal gyrus (MFG: 55b); (vi) inferior frontal gyrus orbital (IFGOrb: a47r, p47r, a9-46v), (vii) posterior cingulate cortex (PCC: 31pv, 31pd, PCV, 7m, 23, RSC); and (viii) dorsal medial prefrontal cortex (dmPFC: 9m, 10d, d32) (Baker et al., 2018; Milton et al., 2021; Desai et al., 2022).

Extracting Word Features from BERT We investigate the alignment with the base pretrained BERT model, provided by Hugging Face (Wolf et al., 2020) (12 layers, 768 dimensions). The primary rationale behind our choice to analyze the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as 'BERTology' (Jawahar et al., 2019; Mohebbi et al., 2021). We follow previous work to extract the hidden-state representations from each layer of pretrained BERT, given a fixed-length input (Toneva & Wehbe, 2019). To extract the stimulus features from pretrained BERT, we constrained the tokenizer to use a maximum context of previous

Sentence Length

TreeDepth

TopConstituents

Tense

Subject Number

Object Number

Sentence Length

TreeDepth

TopConstituents

Tense

Subject Number

Object Number

-1

-0.5

0

0.5

1

1.0

-0.138

0.094

-0.064

0.047

0.002

-0.138

1.0

0.133

0.126

-0.007

0.013

0.094

0.133

1.0

-0.2

0.067

-0.005

-0.064

0.126

-0.2

1.0

-0.136

-0.104

0.047

-0.007

0.067

-0.136

1.0

0.639

0.002

0.013

-0.005

-0.104

0.639

1.0

20 words. Given the constrained context length, each word is successively input to the network with at most C previous tokens. For instance, given a story of M words and considering the context length of 20, while the third word's vector is computed by inputting the network with (w 1 , w 2 , w 3 ), the last word's vectors w M is computed by inputting the network with (w M -20 , . . . , w M ). The pretrained Transformer model outputs word representations at different encoder layers. We use the #words × 768 dimension vector obtained from each hidden layer to obtain word-level representations from BERT. For the analyses using GPT-2, we extract the stimulus features in an identical way from a pretrained GPT-2, provided by Hugging Face (Wolf et al., 2020) (12 layers, 768 dimensions).

Downsampling Since the rate of fMRI data acquisition (TR = 1.5sec) was lower than the rate at which the stimulus was presented to the subjects, several words fall under the same TR in a single acquisition. Hence, we match the stimulus acquisition rate to fMRI data recording by downsampling the stimulus features using a 3-lobed Lanczos filter. After downsampling, we obtain the chunkembedding corresponding to each TR.

TRAlignment To account for the delay of the hemodynamic response, we model the hemodynamic response function using a finite response filter (FIR) per voxel and for each subject separately with 8 temporal delays corresponding to 12 seconds (Nishimoto et al., 2011).

Probing Tasks Probing tasks (Adi et al., 2016; Hupkes et al., 2018; Jawahar et al., 2019) help in unpacking the linguistic features possibly encoded in neural language models. In this paper, we use six popular probing tasks grouped into three categories-surface (sentence length), syntactic (tree depth and top constituents), and semantic (tense, subject number, and object number)-to assess the capability of BERT layers in encoding each linguistic property. Since other popular probing tasks such as Bigram Shift, Odd-Man-Out, and Coordination Inversion (as discussed in Conneau et al. (2018)) have constant labels for our brain dataset, we focused on the remaining six probing tasks mentioned above. Details of each probing task are as follows. Sentence Length: This surface task tests for the length of a sentence. TreeDepth: This syntactic task tests for classification where the goal is to predict depth of the sentence's syntactic tree (with values ranging from 5 to 12). TopConstituents: This syntactic task tests for the sequence of top-level constituents in the syntactic tree. This is a 20-class classification task (e.g. ADVP_NP_NP_, CC_ADVP_NP_VP_). Tense: This semantic task tests for the binary classification, based on whether the main verb of the sentence is marked as being in the present (PRES class) or past (PAST class) tense. Subject Number: This semantic task tests for the binary classification focusing on the number of the subject of the main clause of a sentence. The classes are NN (singular) and NNS (plural or mass: 'colors', 'waves', etc). Object Number: This semantic task tests for the binary classification focusing on the object number in the main clause of a sentence. Details of these tasks are mentioned in Conneau et al. (2018).

Balancing Classes As discussed in Section 4, our method involves removing one linguistic property p at a time. Ideally, removing p should not impact the performance for any other property p ′ . However, if some classes of p co-occur very frequently with a few classes of p ′ , this undesired behavior may happen. The Left column of Appendix Figure 6 shows the presence of such frequent class cooccurrences across properties. To avoid skewed class co-occurrences, we regroup the class labels for three tasks: Sentence Length, TreeDepth and TopConstituents, as shown in Appendix Figure 5.

Probing Task Annotations To annotate the linguistic property labels for each probing task for our dataset, we use Stanford core-NLP stanza library (Manning et al., 2014) for sentence-level annotations. Further, for word-level annotations, we assign the same label for all the words present in a sentence except for the sentence length task.

Probing Tasks Similarity Pearson correlation values between task labels for each pair of tasks were used to construct the similarity matrix for the '21st year' stimulus text, as shown in Figure 2. We observe a high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.

4 Methodology

Removal of Linguistic Properties To remove a linguistic property from the pretrained BERT representations, we use a ridge regression method in which the probing task label is considered as input and the word features are the target. We compute the residuals by subtracting the predicted feature representations from the actual features resulting in the (linear) removal of a linguistic property from pretrained features (see Figure 1 for a schematic). Because the brain prediction method is also a linear function (see next paragraph), this linear removal limits the contribution of the linguistic property to the eventual brain prediction performance.

Specifically, given an input matrix T i with dimension N × 1 for probing task i , and target word representations W ∈ R N × d , where N denotes the number of words (8267) and d denotes the dimensionality of each word (768 dimension), the ridge regression objective function is f ( T i ) = min θ i ∥ W -T i θ i ∥ 2 F + λ ∥ θ i ∥ 2 F where θ i denotes the learned weight coefficient for em- bedding dimension d for the input task i , ∥ . ∥ 2 F denotes the Frobenius norm, and λ > 0 is a tunable hyper-parameter representing the regularization weight for each feature dimension. Using the learned weight coefficients, we compute the residuals as follows: r ( T i ) = W -T i θ i .

We verified that another popular method of removing properties from representations-Iterative Null Space Projection (INLP) (Ravfogel et al., 2020)-leads to similar results (see Appendix Table 4). We present the results from the ridge regression removal in the main paper due to its simplicity.

Similar to the probing experiments with pretrained BERT, we also remove linguistic properties from GPT-2 representations and observe similar results (see Appendix Table 11).

Voxelwise Encoding Model To explore how linguistic properties are encoded in the brain when listening to stories, we use layerwise pretrained BERT features as well as residuals by removing each linguistic property and using them in a voxelwise encoding model to predict brain responses. If a linguistic property is a good predictor of a specific brain region, information about that property is likely encoded in that region. In this paper, we train fMRI encoding models using Banded ridge regression (Tikhonov et al., 1977) on stimulus representations from the feature spaces mentioned above. Before doing regression, we first z-scored each feature channel separately for training and testing. This was done to match the features to the fMRI responses, which were also z-scored for training and testing. The solution to the banded regression approach is given by f ( ˆ β ) = argmin ∥ Y -X β ∥ 2 F + λ ∥ β ∥ 2 F , where Y denotes the voxels matrix across TRs, β denotes the learned

β

regression coefficients, and X denotes stimulus or residual representations. To find the optimal regularization parameter for each feature space, we use a range of regularization parameters that is explored using cross-validation. The main goal of each fMRI encoding model is to predict brain responses associated with each brain voxel given a stimulus.

Cross-Validation The ridge regression parameters were fit using 4-fold cross-validation. All the data samples from K-1 folds were used for training, and the generalization was tested on samples from the left-out fold.

Evaluation Metrics We evaluate our models using Pearson Correlation (PC) which is a popular metric for evaluating brain alignment (Jain & Huth, 2018; Schrimpf et al., 2021; Goldstein et al., 2022). Let TR be the number of time repetitions. Let Y = { Y i } TR i =1 and ˆ Y = { ˆ Y i } TR i =1 denote the actual and predicted value vectors for a single voxel. Thus, Y ∈ R TR and also ˆ Y ∈ R TR . We use Pearson Correlation (PC) which is computed as corr ( Y, ˆ Y ) where corr is the correlation function.

Implementation Details for Reproducibility All experiments were conducted on a machine with 1 NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We used banded ridge-regression with the following parameters: MSE loss function, and L2-decay ( λ ) varied from 10 1 to 10 3 ; the best λ was chosen by tuning on validation data; the number of cross-validation runs was 4.

5 Results

5.1 Successful removal of linguistic properties from pretrained BERT

To assess the degree to which different layers of pretrained BERT representation encode labels for each probing task, a probing classifier is trained over it with the embeddings of each layer. For all probing tasks, we use logistic regression as the probing classifier. We train different logistic regression classifiers per layer of BERT using six probing sentence-level datasets created by Conneau et al. (2018). At test time, for each sample from 21 st year, we extract stimuli representations from each layer of BERT, and perform classification for each of the six probing tasks using the learned logistic regression models. To investigate whether a linguistic property was successfully removed from the pretrained representations, we further test whether we can decode the task labels from the residuals for each probing task of the 21 st year stimuli.

Table 1 reports the result for each probing task, before and after removal of the linguistic property from pretrained BERT. We also report similar results for GPT2 in Table 11 in the Appendix. Similarly to earlier works (Jawahar et al., 2019), we find that BERT embeds a rich hierarchy of linguistic signals also for our 21 st year stimuli: surface information at the initial to middle layers, syntactic information in the middle, semantic information at the middle to top layers. We verify that the removal of each linguistic property from BERT leads to reduced task performance across all layers, as expected.

We further test whether removing information about one property affects the decoding performance for another property. We observed that most properties are not substantially affected by removing other task properties (see Appendix Tables 5, 6, 7, 8, 9 and 10).

5.2 Removal of linguistic properties significantly decreases brain alignment across all layers

In Figure 3 ( left ), we present the average brain alignment across all layers of pretrained BERT before and after the removal of each linguistic property. Removing each linguistic property leads

Random Vector

pretrained BERT

Removal of Sentence Length

Removal of TreeDepth

Removal of TopConstituents

Removal of Tense

Removal of Subject Number

Removal of Object Number

Removal of All Tasks

Removal of Random Task

0

0.05

0.1

Average across all the layers

Avg Pearson Correlation

1

2

3

4

5

6

7

8

9

10

11

12

0.06

0.07

0.08

0.09

0.1

0.11

pretrained BERT

Removal of TopConstituents

Avg Pearson Correlation

to a significant reduction in brain alignment. In contrast, removing a randomly generated linguistic property vector (for this baseline, think about a new task which has no linguistic meaning. The values for each row is then set to a random value chosen uniformly from across all class labels) does not significantly affect the brain alignment. This validates that the reduction in brain alignment observed after removing the linguistic information is indeed due to the elimination of linguistic information, rather than to the removal method itself. Further, we observe that the brain alignment of a random vector is significantly lower than the residual brain alignment after removal of each linguistic property. This suggests that there are additional important properties for the alignment between pretrained BERT and the fMRI recordings, beyond the ones considered in this work.

In Figure 3 (right), we also report the layer-wise performance for pretrained BERT before and after the removal of one representative linguistic property-TopConstituents. We present the results for the remaining properties in Figure 7 in the Appendix. We also report similar results for GPT2 in Figure11 in the Appendix. Similarly to previous work (Toneva & Wehbe, 2019), we observe that pretrained BERT has the best brain alignment in the middle layers. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure). We observe the most substantial reductions in brain alignment in middle to late layers. This pattern holds across all tested linguistic properties. These results provide direct evidence that these linguistic properties in fact significantly affect the alignment between fMRI recordings and pretrained BERT.

Statistical Significance To estimate the statistical significance of the performance differences (across all tasks), we performed a two-tailed paired-sample t-test on the mean correlation values for the subjects. In all cases, we report p-values across layers. For all layers, the main effect of the two-tailed test was significant for all the tasks with p ≤ 0 . 05 with confidence 95% . Finally, the Benjamni-Hochberg False Discovery Rate (FDR) correction for multiple comparisons (Benjamini & Hochberg, 1995) is used for all tests (appropriate because fMRI data is considered to have positive dependence (Genovese, 2000)). Overall, the p-values confirmed that removing each linguistic property from BERT significantly reduced brain alignment.

Figure 10 in the Appendix displays p-values for the brain alignment of pretrained BERT before and after the removal of each linguistic property across all the layers. We observe that the alignment with the whole brain is significantly reduced across all the layers.

ROI-Level Analysis We further examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprehension (Fedorenko et al., 2010;

Fedorenko & Thompson-Schill, 2014) and word semantics (Binder et al., 2009). We find that across language regions, the alignment is significantly decreased by the removal of the linguistic properties across all layers (see Appendix Figure 8). We further investigate the language sub-regions, such as 44, 45, IFJa, IFSp, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Appendix Figure 9).

5.3 Decoding task performance vs brain alignment

While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers. For this purpose, we perform both quantitative and qualitative analyses by measuring the correlation across layers between the differences in decoding task performance from pretrained and residual BERT and the differences in brain alignment of pretrained BERT and residual BERT. A high correlation for a specific linguistic property suggests a strong relationship between the presence of this property across layers in the model and its corresponding effect on the layer-wise brain alignment. Note that this analysis can provide a more direct and stronger claim for the effect of a linguistic property on the brain alignment across layers than alternate analyses, such as computing the correlation between the decoding task performance and the brain alignment across layers only at the level of the pretrained model. We present the results of these analyses for the whole brain and language regions, and language sub-regions in Tables 2 and 3, respectively.

Whole Brain Analysis High correlations in the last column of Table 2 indicate that the syntactic NLP tasks TopConstituents and TreeDepth have the strongest effect on the alignment between BERT and the whole brain. At the level of the whole brain, surface level and semantic properties have a moderate effect on the trend in brain alignment across layers.

ROI-Level Analysis Further, we analyze these correlations for each language brain region separately, and report the results in the remaining columns of Table 2. We make the following observations: (1) Both TreeDepth and TopConstituents are responsible for the bump in brain alignment not just for the entire brain but also for several language ROIs. TopConstituents is important across all language ROIs, but most important in IFG which is related to syntax. This result aligns with previous work that has found the inferior frontal regions to be sensitive to syntax (Friederici et al., 2003; Friederici, 2012). (2) Unlike TopConstituents which has a strong effect on the alignment with all language ROI, TreeDepth has a strong effect only on alignment with the temporal (ATL and PTL) and frontal regions (IFG and MFG). (3) Although semantic properties are not shown to be responsible for the shape in brain alignment at the whole-brain level, semantic properties such as Tense, Subject Number and Object Number affect the alignment more specifically in PCC. This finding agrees with previous work which suggests the PCC supports word semantics (Binder et al., 2009). (4) The surface property

Removal of Subject Number

Removal of Tense

L/R

Removal of TopConstitue

n

ts

Removal of

Sentence

Length

Removal of TreeDepth

Removal of Object Number

L/R

L/R

L/R

L/R

L/R

0.6

t'0

0.4

0.4

0.2

0.4

0.4

0.2

Sentence Length has a strong effect on the alignment with the IFG and MFG, but not as high as the syntactic properties in these language ROIs.

Sub-ROI-Level Analysis Each language brain region is not necessarily homogeneous in function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. Thus, we further analyze several important language sub-regions that are thought to exemplify the variety of functionality across some of the broader language regions (Rolls et al., 2022): AG sub-regions (PFm, PGi, PGs), ATL sub-regions (STGa, STSda), PTL sub-regions (STSdp, A5, TPOJ1, PSL, STV, SFL), and IFG sub-regions (44, 45, IFJa, IFSp). We present the correlations for each of these sub-regions separately in Table 3. We make the following observations: (1) Although TopConstituents strongly affect the trend in the brain alignment across layers for the IFG, it strongly affects the trend for only three sub-regions (44, 45, and IFSp). It is possible that other factors affect the trend in alignment with IFJa across layers, as this IFG region is involved in many cognitive processes, including working memory for maintaining and updating ongoing information (Rainer et al., 1998; Zanto et al., 2011; Gazzaley & Nobre, 2012). (2) While TopConstituents was shown to have the strongest effect on the brain alignment with the whole IFG, the alignment with two of the IFG sub-regions (IFSp and IFJa) is more strongly affected by Tree Depth. (3) Subject Number has a medium to strong effect on the alignment with AG sub-region PFm. This implies that the semantic property Subject Number is important in at least one sub-region of language ROI. (4) Similarly, we find Object Number to be important for the trend of alignment across layers with most language sub-regions of ATL and PTL. Taken together, these results show that the semantic properties are most responsible for the trend in brain alignment across layers for many language sub-regions of ATL, PTL and AG, while syntactic properties have the highest effect on the alignment with the IFG.

Qualitative Analysis To present the correlations above at an even finer grain, we show them now at the voxel-wise level in Figure 4. We make the following qualitative observations: (1) These results confirm that Sentence Length and Subject Number are the least related to the pattern of the brain alignment across layers. (2) The effect of Tree Depth, Top Constituents, and Object Number is more localized to the canonical language regions in the left hemisphere and is more distributed in the right hemisphere. (3) The semantic property Object Number has a much larger effect on the alignment with the left hemisphere. Furthermore, Appendix Figure 12 shows the voxel-wise correlations across layers when removing all linguistic properties, instead of each individual one. We observe that removing all linguistic properties leads to very different region-level brain maps compared to removing individual

linguistic properties (Figure 4), and that the remaining correlation across layers after removing all linguistic properties is not substantial in the key language regions.

6 Discussion and Conclusion

We propose a direct approach for evaluating the joint processing of linguistic properties in brains and language models. We show that the removal of a range of linguistic properties from both language models (pretrained BERT and GPT2) leads to a significant decrease in brain alignment across all layers in the language model.

To understand the contribution of each linguistic property to the trend of brain alignment across layers, we leverage an additional analysis for the whole brain, language regions and language sub-regions: computing the correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. For the whole brain, we find that Tree Depth and Top Constituents are the most responsible for the trend in brain alignment across BERT layers. Specifically, TopConstituents has the largest effect on the trend in brain alignment across BERT layers for all language regions, whereas TreeDepth has a strong effect on a smaller subset of regions that include temporal (ATL and PTL) and frontal language regions (IFG and MFG). Further, although other properties may not have a large impact at the whole brain level, they are important locally. We find that semantic properties, such as Tense, Subject Number, and Object Number affect the alignment with more semantic regions, such as PCC (Binder et al., 2009).

One limitation of our removal approach is that any information that is correlated with the removed linguistic property in our dataset will also be removed. This limitation can be alleviated by increasing the dataset size, which is becoming increasingly possible with new releases of publicly available datasets. It is also important to note that while we find that several linguistic properties affect the alignment between fMRI recordings and a pretrained language model, we also observed that there is substantial remaining brain alignment after removal of all linguistic properties. This suggests that our analysis has not accounted for all linguistic properties that are jointly processed. Future work can build on our approach to incorporate additional linguistic properties to fully characterize the joint processing of information between brains and language models. The current study was performed using experimental stimulus in English and language models trained on English text. While we expect our results to be generalizable at least to languages that are syntactically close to English, this should be explored by future work. Additionally, investigating larger language models beyond the two models used in our study, BERT and GPT-2, can contribute to an even more comprehensive understanding of the reasons for alignment between language models and human brain activity.

The insights gained from our work have implications for AI engineering, neuroscience and model interpretability-some in the short-term, others in the long-term. AI engineering: Our work most immediately fits in with the neuro-AI research direction that specifically investigates the relationship between representations in the brain and representations learned by powerful neural network models. This direction has gained recent traction, especially in the domain of language, thanks to advancements in language models (Schrimpf et al., 2021; Goldstein et al., 2022). Our work most immediately contributes to this line of research by understanding the reasons for the observed similarity in more depth. Specifically, our work can guide linguistic feature selection, facilitate improved transfer learning, and help in the development of cognitively plausible AI architectures. Computational Modeling in Neuroscience: Researchers have started viewing language models as useful model organisms for human language processing (Toneva, 2021) since they implement a language system in a way that may be very different from the human brain, but may nonetheless offer insights into the linguistic tasks and computational processes that are sufficient or insufficient to solve them (McCloskey, 1991; Baroni, 2020). Our work enables cognitive neuroscientists to have more control over using language models as model organisms of language processing. Model Interpretability: In the long-term, we hope that our approach can contribute to another line of work that uses brain signals to interpret the information contained by neural network models (Toneva & Wehbe, 2019; Aw & Toneva, 2023). We believe that the addition of linguistic features by our approach can further increase the model interpretability enabled by this line of work. Overall, we hope that our approach can be used to better understand the necessary and sufficient properties that lead to significant alignment between brain recordings and language models.

Acknowledgments

This work was partially funded by the German Research Foundation (DFG) - DFG Research Unit FOR 5368. We thank Gabriele Merlin and Camila Kolling for providing valuable feedback on earlier versions of this work.

References

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207 , 2016.

Andrew J Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. TACL , 5:17-30, 2017.

Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations , 2023.

Cordell M Baker, Joshua D Burks, Robert G Briggs, Andrew K Conner, Chad A Glenn, Kathleen N Taylor, Goksel Sali, Tressie M McCoy, James D Battiste, Daniel L O'Donoghue, et al. A connectomic atlas of the human cerebrum-chapter 7: the lateral parietal lobe. Operative Neurosurgery , 15(suppl_1):S295-S349, 2018.

Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society B , 375(1791):20190307, 2020.

Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57(1):289-300, 1995.

Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex , 19(12):2767-2796, 2009.

Charlotte Caucheteux and Jean-Rémi King. Language processing in brains and deep neural networks: computational convergence and its limits. BioRxiv , 2020.

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single! vector: Probing sentence embeddings for linguistic properties. In ACL 2018-56th Annual Meeting of the Association for Computational Linguistics , volume 1, pp. 2126-2136. Association for Computational Linguistics, 2018.

Rutvik Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system, 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Fedorenko, P.-J. Hsieh, A. Nieto-Castanon, S. Whitfield-Gabrieli, and N. Kanwisher. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Neurophysiology , 104(2):1177-1194, 2010.

Evelina Fedorenko and Sharon L Thompson-Schill. Reworking the language network. Trends in cognitive sciences , 18(3):120-126, 2014.

Angela D Friederici. The cortical language circuit: from auditory perception to sentence comprehension. Trends in cognitive sciences , 16(5):262-268, 2012.

Angela D Friederici, Shirley-Ann Rüschemeyer, Anja Hahne, and Christian J Fiebach. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cerebral cortex , 13(2):170-177, 2003.

Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.

Adam Gazzaley and Anna C Nobre. Top-down modulation: bridging selective attention and working memory. Trends in cognitive sciences , 16(2):129-135, 2012.

Christopher R Genovese. A bayesian time-course model for functional magnetic resonance imaging data. Journal of the American Statistical Association , 95(451):691-703, 2000.

Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature , 536(7615):171-178, 2016.

Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience , 25(3):369-380, 2022.

Olaf Hauk and Friedemann Pulvermüller. Effects of word length and frequency on the human event-related potential. Clinical Neurophysiology , 115(5):1090-1103, 2004.

Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang. Cognival: A framework for cognitive word embedding evaluation. In CoNLL , pp. 538-549, 2019.

Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research , 61:907-926, 2018.

Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fMRI. In Advances in Neural Information Processing Systems , pp. 6628-6637, 2018.

S Jat, H Tang, P Talukdar, and T Mitchel. Relating simple sentence representations in deep neural networks and the brain. In ACL , pp. 5137-5154, 2020.

Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics , 2019.

Sreejan Kumar, Theodore R Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. bioRxiv , 2022.

Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations , pp. 55-60, 2014.

Michael McCloskey. Networks and theories: The place of connectionism in cognitive science. Psychological science , 2(6):387-395, 1991.

Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word-level semantics and prediction. arXiv preprint arXiv:2212.00596 , 2022.

Camille K Milton, Vukshitha Dhanaraj, Isabella M Young, Hugh M Taylor, Peter J Nicholas, Robert G Briggs, Michael Y Bai, Rannulu D Fonseka, Jorge Hormovas, Yueh-Hsin Lin, et al. Parcellationbased anatomic model of the semantic network. Brain and behavior , 11(4):e02065, 2021.

Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. Exploring the role of bert token representations to explain sentence probing results. In The 2021 Conference on Empirical Methods in Natural Language Processing , pp. 792-806. Association for Computational Linguistics, 2021.

Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The 'narratives' fmri dataset for evaluating models of naturalistic language comprehension. Scientific data , 8(1): 1-22, 2021.

Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology , 21(19):1641-1646, 2011.

Subba Reddy Oota, Vijay Rowtula, Manish Gupta, and Raju S Bapi. Stepencog: A convolutional lstm autoencoder for near-perfect fmri encoding. In IJCNN , pp. 1-8. IEEE, 2019.

Subba Reddy Oota, Frederic Alexandre, and Xavier Hinaut. Long-term plausibility of language models and neural dynamics during narrative listening. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 44, 2022a.

Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? NAACL , 2022b.

Subba Reddy Oota, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampud. Syntactic structure processing in the brain while listening. Findings of the Association for Computational Linguistics: ACL , 2023.

Christophe Pallier, Anne-Dominique Devauchelle, and Stanislas Dehaene. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences , 108(6): 2522-2527, 2011.

Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Decoding of generic mental representations from functional mri data using word embeddings. bioRxiv , pp. 057216, 2016.

Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature communications , 9(1):1-13, 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Gregor Rainer, Wael F Asaad, and Earl K Miller. Memory fields of neurons in the primate prefrontal cortex. Proceedings of the National Academy of Sciences , 95(25):15008-15013, 1998.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. Association for Computational Linguistics , 2020.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics , 8:842-866, 2020.

Edmund T Rolls, Gustavo Deco, Chu-Chung Huang, and Jianfeng Feng. The human language effective connectome. NeuroImage , pp. 119352, 2022.

Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences , 118(45), 2021.

Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. Advances in neural information processing systems , 32, 2019.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Towards sentence-level brain decoding with distributed representations. In AAAI , pp. 7047-7054, 2019.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Neural encoding and decoding with distributed sentence representations. IEEE Transactions on Neural Networks and Learning Systems , 32(2):589-603, 2020.

Andre˘ ı Nikolaevich Tikhonov, Vasilij Ja Arsenin, and Vasili˘ ı Arsenin. Solutions of ill-posed problems . V H Winston, 1977.

Mariya Toneva. Bridging Language in Machines with Language in the Brain . PhD thesis, Carnegie Mellon University, 2021.

Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in Neural Information Processing Systems , 32, 2019.

Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science , 2(11):745-757, 2022.

Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. Fine-grained neural

decoding with distributed word representations. Information Sciences , 507:256-272, 2020.

Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , pp. 233-243, 2014.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pp. 38-45, 2020.

Theodore P Zanto, Michael T Rubens, Arul Thangavel, and Adam Gazzaley. Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory. Nature neuroscience , 14(5):656-661, 2011.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile several papers have recently shown that large language model embeddings are aligned with / predictive of fMRI reading data, this paper pushes these analyses forward by asking: what properties of the representations are responsible for this alignment?  In particular, they compare LLM/fMRI alignment on reading data using BERT, but then compare this to BERT's representations _when certain linguistic properties have been removed_ (using both a novel and an existing removal method).  The results show a significant decrease in alignment (crucially, not found when removing \"random\" information or when using random vectors) across all linguistic properties, suggesting that it really is BERT's encoding of linguistic information that's responsible for the alignment with fMRI reading data.  This represents a finer-grained level of analysis than existing alignment works and should be of interest to many in the field.\n\nSTRENGTHS:\n- Analyzes LLM representations' ability to predict fMRI data before and after \"removing\" linguistic information from said representations.\n- Conducts analyses at both macro (whole-model, and whole-brain) as well as micro (layer-wise, ROI and sub-ROI) levels.\n\nWEAKNESSES:\n- The Pearson correlations in Figure 3 are quite low, on average less than 0.1.  What do the authors make of this?  The ROI-level analyses seem relevant, but are not done with the same metric, so direct comparison is hard.\n- Linguistic annotations for the text in the reading data come from an off-the-shelf NLP tool (stanza), so these are \"silver\" rather than \"gold\" annotations.\n- More baselines would have been useful.\n\nQUESTIONS:\n- Table 1: what are baselines for each task?  What would a most-frequent-class baseline score on each?  This would be a helpful comparison for the post-removal numbers.\n- Word length probing: I'm a bit confused here.  Probing for word length should be hard for BERT since it does not have character-level information but instead uses sub-word tokenization.  I noticed in the Appendix that the task is referred to as \"Sentence Length\".  Is the task length-of-individual-word (that's what \"Word Length\" sounds like to me) or is it number-of-words-in-sentence (\"Sentence Length\")?  Consistency with naming, and clarity on the exact task, would be welcome here.\n- Why the choice of BERT as the model for comparison?  In particular, I am curious about the choice of a \"bidirectional\" masked language model as opposed to a \"left-to-right\" language model like the GPT series.  Some would argue that the latter more closely resemble the reading process, and so might also better predict data from reading.\n\n\n### Typographic comments:\n\n- Line 125 needs a closing period.\n- Lines 185 and 186: \"z-scored\" -> \"$z$-scored\".  Same for \"t-test\" and \"p-value\" throughout the paper.\n\nLIMITATIONS:\n- Use of stanza corenlp for linguistic annotations; these are \"silver\", not \"gold\".\n- Only looked at one model, with relatively weak overall correlations with brain activity.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper aims to investigate the correspondence between HIPSs and models based on Neural Networks such as the transformers. The key innovation of this paper is to study this correspondence by elimiating specicific linguistic properties form BERT and to observe how this intervation affects the alignment with fMRI brain recording.

STRENGTHS:
- This is really an innovative standpoint

WEAKNESSES:
- There is a major innovation of this paper that is the cornerstone of the theory. Yet, this is not listed as a major result: the removal of linguistic properties from Transformers. Indeed, I was expecting citations to other papers as this is not listed in the main results of the paper. Yet, there are not citations and it seems to be proposed in this paper. 
 

- Since the removal of linguistic properties from Transformers is a major cornerstone, all the theory could crumble if it is not investigated in a deep and correct way.

- Experiments to demonstrate that the removal of linguistic properties from Transformers are inconclusive. Indeed, Table 1 is not sufficient. It only says that after the adpatation BERT weights are so mess up that the classification task cannot be performed. There should be a comparative part or a sort of proof that the model is not compromised in the other parts. For example, there is a missing column: BERT with random parameters. Indeed, if the "after" models behave as BERT with random inizialization, results are not relevant.

QUESTIONS:
see weaknesses

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the relationship between specific linguistic properties and brain alignment across layers of a language model. The authors provide evidence for the importance of a linguistic property to the brain alignment. The focus of the study is to understand the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.\n\nSTRENGTHS:\n**Originality:** The paper offers an interesting perspective on the link between language properties and brain alignment in language models. It builds on existing work but adds a new dimension by directly showing the importance of specific linguistic properties.\n\n**Quality:** The paper uses a large, public fMRI dataset and word features from the BERT model to get word-level representations. The results are interesting. \n\n**Clarity:** The paper is well-structured and easy to understand. It explains the methods and results clearly, making it accessible to a wide range of readers.\n\n**Significance:** The approach paves a path towards a way to understand how linguistic properties and brain alignment relate. This could help improve language models and give new insights in neuroscience and language processing.\n\nWEAKNESSES:\n**Choice of Language Model:** The authors have chosen to focus on BERT, an encoder-only language model, for their study. Given that previous research has shown a deeper connection between autoregressive language models and the brain, the choice of BERT might limit the generalizability of the findings. The authors could consider including autoregressive language models in their analysis to provide a more comprehensive understanding of the relationship between linguistic properties and brain alignment.\n\n**Correlation vs. Causation:** The authors aim to identify the effect of linguistic properties on brain alignment, but they rely on correlational metrics, which do not provide evidence of causation. To address this, the authors could consider methods from the causality+NLP literature to produce causal model explanations. This would strengthen their findings and provide more definitive evidence of the impact of linguistic properties on brain alignment. As it currently stands, it’s unclear which conclusions can be drawn from this study.\n\n**Projection Method:** The authors use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). The authors could consider alternative methods for identifying the effect of linguistic information to ensure the accuracy and reliability of their findings. Some potential approaches include causal abstractions, which aim to provide a low-dimensional causal representation of LLMs.\n\nQUESTIONS:\n1. **Choice of Language Model:** Could you elaborate on why you chose to focus only on BERT, an encoder-only language model, for your study? Given that previous research has shown a deeper connection between autoregressive language models and the brain, have you considered including autoregressive language models in your analysis? How do you think the inclusion of such models might affect your findings?\n\n2. **Correlation vs. Causation:** Your study aims to identify the effect of linguistic properties on brain alignment, but it relies on correlational metrics. Could you consider methods from the Causality+NLP literature to produce causal model explanations? How do you think a causal analysis might change your findings or conclusions?\n\n3. **Projection Method:** You use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). Could you consider alternative methods for identifying the effect of linguistic information? How do you think a different method might affect your findings?\n\nLIMITATIONS:\nThe authors do recognize some important limitations:\n\n1. **Removal Approach:** They note that their method might unintentionally remove information correlated with the linguistic property they're studying. They suggest using larger datasets to mitigate this.\n\n2. **Unaccounted Linguistic Properties:** They admit that there's still significant brain alignment even after removing all studied linguistic properties, indicating that there might be other linguistic properties at play that they haven't considered.\n\n3. **Single Model Use:** They acknowledge that they've only tested their findings on one language model (BERT) and suggest that future work should test other models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide an analysis of correlations between BERT representations and fMRI data when abstracting away different linguistic properties from the BERT representations. They provide an in-depth analysis of the ways in which elimination of word length, tree depth, top constituents, tense, subject number and object number have on the overall and layer-wise alignment of BERT with fMRI data from 18 people. The authors find that removing any of these linguistic properties from the BERT representations reduces fMRI data alignment across all layers in the BERT model. Furthermore, syntactic properties are most relevant for the BERT-brain alignment globally, but semantic properties are highly locally aligned with brain regions generally associated with semantic processing.\n\nSTRENGTHS:\nThe overall question and method is relevant to the community. This paper contributes to the cognitive debate on the overall alignment that today's language models have with people's cognitive systems, here concretely their linguistic neural processing of stories. By abstracting away a number of linguistic properties from the BERT representations in a controlled way, the results speak for the type of information encoded in BERT that lead to the reasonably high alignment with fMRI data. The work therefore contributes to a better understanding of BERT representations and the usefulness of investigating them as cognitive systems.\n\nThe authors integrate valuable controls into their study, such as random baselines and correlational analyses of tasks that nicely contextualize the findings.\n\nThe paper is very well-written.\n\nWEAKNESSES:\nTo me, section 5.3 (Decoding task performance vs. brain alignment) has a number of interesting observations but I'm missing the overall motivation for this section. The authors write \"While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers.\" (line 265ff). There seems to be a close connection between this question and Figure 3a from the previous section and it would be helpful to emphasize what the previous section lacks. To be clear, I think section 5.3 adds a lot, especially when it comes to the alignment of local instead of global brain regions but I currently can't find a motivation for whom this analysis is most relevant -- does this tell us something about BERT or are there any potential insights for neuroscience? The semantic alignment results seem to be backed up by prior findings in neuroscience. Is the same true for the rest, or if not, what does that mean?\n\nBERT's layer-wise analysis of the correlations with fMRI data are central in the overall framing and centered in the main paper. I think that's rightfully so but I would like to encourage the authors to consider a strategy of substantiating claims that \"BERT embeds a rich hierarchy [with] surface information at the initial and middle layers, syntactic information in the middle to top layers [...]\" (lines 218ff). From inspecting the corresponding results table (Table 1), surface information seem maybe slightly more represented in the first and least in the second layer, but all other layers seem to be fairly evenly distributed. For the syntactic information, it seems uniformly high, and only subject number and object number seem to have a fairly monotonously increasing and decreasing structure that might suggest that it can't simply be due to random variance. Is there any way to estimate the variance of these terms or find a definition for when something is robustly represented more in some than other layers?\n\nQUESTIONS:\nI find Figure 3 very intriguing. Removing any linguistic property results in a correlation drop from 0.1 to about 0.75. The authors also previously added a correlational analysis indicating that most linguistic properties aren't correlated with each other. Consequently, it would make sense to me that when one removes all linguistic properties, the correlation needs to drop further since different sources of variance have been removed which don't correlate with each other but clearly correlate with the fMRI data. However, even this correlation is at about 0.75. Do you have an explanation why this might be?\n\nPicking up on a notion in the \"Weaknesses\" section: Why doesn't figure 3 already provide us with an answer to the central motivation of section 5.3 (\"we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers.\")? Or did you mean to motivate section 5.3 with \"across brain regions\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}