Can fMRI reveal the representation of syntactic structure in the brain?

1 Introduction . Neuroscientists have long been interested in how the brain processes syntax . To date , there is no consensus on which brain regions are involved in processing it . Classically , only a small number of regions in the left hemisphere were thought to be involved in language processing . More recently , the language system was proposed to involve a set of brain regions spanning the left and right hemisphere [ 1 ] . Similarly , some findings show that syntax is constrained to specific brain regions [ 2 , 3 ] , while other findings show syntax is distributed throughout the language system [ 4–6 ] . The biological basis of syntax was first explored through studies of the impact of brain lesions on language comprehension or production [ 7 ] and later through non-invasive neuroimaging experiments that record brain activity while subjects perform language tasks , using methods such as functional Magnetic Resonance Imaging ( fMRI ) or electroencephalography ( EEG ) . These experiments usually isolate syntactic processing by contrasting the activity between a difficult syntactic condition and an easier one and by identifying regions that increase in activity with syntactic effort [ 3 ] . An example of these conditions is reading a sentence with an object-relative clause ( e.g . “ The rat that the cat chased was tired '' ) , which is more taxing than reading a sentence with a subject-relative clause ( e.g . “ The cat that chased the rat was tired '' ) . In the past decade , this approach was extended to study syntactic processing in naturalistic settings such as when reading or listening to a story [ 8–11 ] . Because such complex material is not organized into conditions , neuroscientists have instead devised 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . complexity metrics capturing the word-by-word evolving syntactic demands required to understand the material . Using these metrics , neuroscientists were able to identify regions with activity correlated with syntactic processing load , and suggest the involvement of these regions in syntactic processing . While many have studied syntactic processing as captured through complexity measures , very few have studied the syntactic representations themselves . As an analogy , consider the neurobiology of language semantics . A large part of the literature has focused on characterizing the semantic processing load related to integrating or predicting incoming words ( see [ 12 ] for a review ) . Concurrently , another part has focused on studying where the meaning of words itself is represented , either through contrast-based studies ( see [ 13 ] for a review ) or through encoding model approaches ( e.g . [ 14–19 ] ) . It can be argued however , that in syntax , most research focuses on the first approach ( using complexity ) and not the second ( using representations ) . There are two main reasons why the study of syntactic representations using fMRI is difficult : 1 . The first reason is computational . To identify the brain correlates of syntactic representations , one has to embed the syntactic representation ( often a tree ) into a vector that can be then used to predict the time series of brain activity as a function of syntactic structure . This vector representation should change as the words are processed incrementally . The construction of such a vector space is akin to the problem of building a graph embedding , and it is not trivial . The goal is to have different sentences or segments of sentences with similar structure ( irrespective of their meanings ) map to nearby points in the vector space . 2 . The second reason is that the fMRI signal is noisy , and it is not clear that the neural basis of the representation of syntactic structure lends itself to being studied using fMRI . For instance , it could be that the neural substrate for syntax is intermingled with that of other language components like semantics and is hard to disentangle , or it could be that neurons inside the same voxel perform different syntactic computations , making it hard to differentiate the signal corresponding to these different computations . More generally , it could also be that syntactic computations are organized in a way that the low Signal-to-Noise Ratio ( SNR ) of fMRI makes it difficult to study them . To help address the first difficulty , we propose syntactic structure embeddings that encode the syntactic information inherent in natural text that subjects read in the scanner . These structure embeddings are proposed as an additional tool in the arsenal of neurolinguists and can serve to ask the question of whether the fMRI signal can reveal syntactic representations . We use our syntactic structure embeddings in a voxelwise encoding model framework [ 14 , 16 , 17 , 20 ] . We find that our embeddings – along with other simpler syntactic structure embeddings built using conventional syntactic features such as part-of-speech ( POS ) tags and dependency role ( DEP ) tags – are able to explain an additional portion of the variance in the fMRI data of subjects reading text , even after controlling for complexity metrics that capture processing load , suggesting that fMRI can indeed reveal syntactic representations . Furthermore , the regions that are well-predicted by syntactic representations are distributed across the language network . We also attempt to minimize the amount of semantic information present in our embeddings . After showing that they do not encode a significant amount of semantics , we use our syntactic embeddings to address the aforementioned issue of whether regions that are predicted by syntactic features are selective for syntax , meaning they are only responsive to syntax and not to other language properties such as semantics . To answer this question , we use a contextual word embedding space [ 21 ] that integrates semantics and syntax . Consistent with prior literature , regions that are predicted by syntax are much better predicted by the contextual embeddings and do not appear to be selective for syntax . 2 Background . Naturalistic tasks Naturalistic tasks are gradually becoming the paradigm of choice to study language processing in the brain . Many have used naturalistic listening tasks to investigate syntactic processing by measuring word-by-word processing load [ 8–10 , 22–25 ] . Even though story reading or listening involves multiple types of processing ( low-level processing , word recognition , building sentences , inferring meaning and social reasoning , etc . ) , researchers are gradually learning more and more about how these processes are mapped to different brain areas . It has already been shown that the regions defined by Fedorenko et al . [ 26 ] as the language network ( also used in this work ) process word-level syntactic [ 16–18 ] and semantic features as well as sequence meaning [ 19 , 27 , 28 ] . Processing information beyond the level of sentences is thought to recruit other areas [ 17 , 29 ] . Encoding models are frequently used to analyze data collected using naturalistic tasks . These are models that are trained to predict brain activity at every voxel as a function of some feature of interest . Voxels that are well-predicted are thought to be involved in processing the information encoded in the feature . The various complexity metrics were formulated as features that encode syntactic demands . Complexity metrics Several complexity metrics have been proposed to capture word-by-word syntactic demands . One common approach for constructing a syntactic complexity metric is to assume a sentence ’ s syntactic representation and estimate the number of syntactic operations performed at each word . Node Count is a popular metric constructed using this approach [ 30 ] . It relies on constituency trees ( structures that capture the hierarchical grammatical relationship between the words in a sentence ) . While traversing the words of the sentence in order , subtrees of this tree get completed ; Node Count refers to the number of such subtrees that get completed at each word , effectively capturing syntactic load or effort . Important work in neurolinguistics [ 8 ] has used Node Count to support the theory that the Anterior Temporal Lobe ( ATL ) is involved in syntactic processing . Complexity metrics are a good proposal for identifying brain regions involved in processing the syntactic structure of a sentence . They can be used to study specific hypotheses about how syntactic structure is built . For example , one could compute these metrics using different types of parsers to determine which parser ’ s operations are more similar to the brain ’ s structure building processes [ 22 ] . Using these metrics also leads to more robust results when compared to those obtained by contrasting syntactic conditions . This is because of two reasons . First , the naturalistic stimuli used in such studies are very diverse . Second , since the metrics can be computed at every word ( i.e . they are incremental metrics ) , the correlation between syntactic processing load and the recorded brain activity can be robustly estimated using a large number of data points ( i.e . words ) . Representing syntactic structure The focus of this work is to study syntactic representations in the brain by building features that encode a sentence ’ s syntactic structure and using them in voxelwise encoding models . Constituency trees and dependency trees are the two main structures that capture a sentence ’ s syntactic structure . Constituency trees are derived using phrase structure grammars ( PSGs ) that encode valid phrase and clause structure ( see fig . 1 ( A ) for an example ) . Dependency trees encode relations between pairs of words such as subject-verb relationships . We derive representations from both types of trees . We derive word-level DEP tags from dependency trees , and encode the structural information given by constituency trees to capture the hierarchical representations of phrase structure . We characterize the syntactic structure inherent in sentence constituency trees by computing an evolving vector representation of the syntactic structure processed at each word using the subgraph embedding algorithm by Adhikari et al . [ 31 ] . We resort to developing this new embedding pipeline because we want our constituency tree-based embeddings to be almost purely syntactic . One might suggest using the internal representations of a neural parser ( such as the parser used by Hale et al . [ 9 ] ) to directly encode syntax . However , neural parsers use word embeddings , making it likely for there to be some semantics in the internal representations . This `` leakage '' of semantics makes it very difficult to isolate the brain regions that process syntax . The problem is further compounded by previous findings which indicate that syntax and semantics might be processed in the same regions of the brain [ 6 ] , making it impossible to infer whether a region ’ s activity is correlated with the syntax or the semantics encoded in the features . 3 Methods . We first describe the syntactic features used in this study and their generation . All of the features we use are incremental i.e . they are computed per word . We then describe our fMRI data analyses . Complexity metrics We use four complexity metrics in our analyses - Node Count , Syntactic Surprisal , word frequency and word length . To compute Node Count , we obtain the constituency tree of each sentence using the self-attentive encoder-based constituency parser by Kitaev and Klein [ 32 ] . We compute Node Count for each word as the number of subtrees that are completed by incorporating this word into its sentence . Syntactic Surprisal is another complexity metric proposed by Roark et al . [ 33 ] and is computed using an incremental top-down parser [ 34 ] . It measures how unexpected it is to read a given word in the current syntactic context . Both of these metrics aim to measure the amount of effort that is required to integrate a word into the syntactic structure of its sentence . The word frequency metric is computed using the wordfreq package [ 35 ] as the Zipf frequency of a word . This is the base-10 logarithm of the number of occurrences per billion of a given word in a large text corpus . Finally , word length is the number of characters in the presented word . The last two metrics approximate the amount of effort that is required to read a word , irrespective of its context . Constituency Tree-based Graph Embeddings ( ConTreGE ) Constituency trees are a rich source of syntactic information . We build three representations of these trees that encode this information : 1 . The largest subtree which is completed upon incorporating a word into a sentence ( see fig . 1 ( A ) ) is representative of the implicit syntactic information given by the word . A subtree is considered complete when all of its leaves are terminals . The largest subtree completed by a given word refers to the subtree with the largest height that also satisfies the following conditions - the given word must be one of its leaves and all of its leaves must only contain words that have been seen till then . Given that Node Count reduces all of the information present in these subtrees to just one number , it is easy to see that it can not fully capture this information . POS tags ( categorize words into nouns , verbs , adjectives , etc . ) also capture some of the information present in these trees as they encode lexical syntactic information . But , they are incapable of completely encoding their hierarchical structure and the parsing decisions made while generating them . To better encode their structure , we first build subgraph embeddings of these completed subtrees called ConTreGE Comp vectors . 2 . We hypothesize that the brain not only processes structure seen thus far but also predicts future structure from structure it already knows . To test this , we construct embeddings , called ConTreGE Incomp vectors , using incomplete subtrees that are constructed by retaining all the PSG productions that are required to derive the words seen till then , starting from the root of the sentence ’ s tree . A non-terminal node is expanded only if it eventually leads to the derivation of a word that has been seen . Any other non-terminal nodes are not expanded . Encoding such subtrees allows us to capture future sentence structure ( in the form of future constituents ) before the full sentence is read ( see fig . 1 ( B ) ) . These subtrees contain leaves that are non-terminal symbols unlike complete subtrees that only have terminal symbols ( words and punctuation ) as leaves . In this context , a non-terminal symbol is a symbol that can be derived further using some rule in the PSG ( e.g . NP , VP , etc. ) . If incomplete subtrees are more representative of the brain ’ s processes , it would mean that the brain correctly predicts certain phrase structures even before the entire phrase or sentence is read . ConTreGE Comp and ConTreGE Incomp vectors need to be built using accurate constituency trees constructed using the whole sentence . Thus , we reuse the trees generated to compute Node Count to build them . 3 . Further , the brain could be computing several possible top-down partial parses that can derive the words seen thus far ( see fig . 1 ( C ) ) and modifying the list of possible parses as future words are read . To test this hypothesis , we designed Incremental ConTreGE ( InConTreGE ) vectors that are representative of the most probable parses so far . For a given word , its InConTreGE vector is computed as : v = ∑5 i=1 e −siWi where Wi is the subgraph embedding of a partial parse tree built by an incremental top-down parser [ 34 ] after reading the word and si is the score assigned to this partial parse that is inversely proportional to the parser ’ s confidence in this tree . The InConTreGE feature space is constructed to encode the different possible parse trees that can derive the words seen so far . These different parse trees try to encode uncertainty about the parent nodes of the words seen so far . They also do not encode any information about possible future children of these parent nodes . However , with ConTreGE Incomp , we encode parts of the final parse tree of a sentence . It does not encode any uncertainty . With this embedding space we are trying to measure the ability of the brain to correctly predict future constituents . We see uncertainty about the current parse and the ability to correctly predict future constituents as two related but distinct concepts . This is the reason why we use the ConTreGE Incomp to hypothesize about the brain encoding correct future information . To effectively capture the structure of all subtrees , we encode them using the subgraph embeddings proposed by Adhikari et al . [ 31 ] which preserve the neighbourhood properties of subgraphs . A long fixed length random walk on a subgraph is generated to compute its embedding . Since consecutive nodes in a random walk are neighbours , a long walk can inform us about the neighbourhoods of nodes in the subgraph . Each node in a walk is identified using its unique ID . So , a random walk can be interpreted as a “ paragraph '' where the words are the node IDs . Finally , the subgraph ’ s embedding is computed as the Paragraph Vector [ 36 ] of this paragraph that is representative of the subgraph ’ s structure . Note that all of the subtrees of a given type ( complete , incomplete or partial parse ) are encoded together , ensuring that all ConTreGE Comp , all ConTreGE Incomp and all InConTreGE vectors are in their own spaces . Fig . 2 illustrates the subtree encoding process . First , every unique non-terminal in the subtrees is mapped to a unique number ( e.g . S is mapped to 1 , NP is mapped to 2 , etc . ) and every terminal is mapped to a unique number that is representative of the order in which they were presented ( the first presented token is mapped to 10000 , the second token is mapped to 10001 and so on ) . We did not map each unique terminal to a unique number ( for instance , we did not map all instances of `` Harry '' to one number ) because a random walk through the tree could give us word co-occurrence information and thus lead to the inclusion of some semantics in the vectors . Every tree node ’ s label is then replaced by the number it was mapped to in the previous step . The edge lists of these subtrees are supplied to the subgraph embedding generation algorithm to finally obtain 15-dimensional vectors for every presented word . The length of the random walks is set to 100000 and we use an extension of the Distributed Bag of Words ( DBOW ) model proposed by Le and Mikolov [ 36 ] for generating Paragraph Vectors called Sub2Vec-DBON by Adhikari et al . [ 31 ] . The sliding window length is set to 5 and the model is trained for 20 epochs . Since ConTreGE Comp , ConTreGE Incomp and InConTreGE encode information about the neighbourhoods of all nodes in the constituency trees , they can capture their hierarchical structure . Thus , brain regions predicted by these vectors are likely to be involved in building and encoding hierarchical sentence structure . Punctuation We create one-hot binary vectors indicating the type of punctuation that was presented along with a word ( e.g. . or , ) . For example , a sentence might have ended with `` Malfoy. '' . In this punctuation-based feature space , the column corresponding to . will be set to 1 for this word . While punctuation is seldom considered a syntactic feature , sentence boundaries are highly correlated with changes in working memory load . These changes are bound to be a great source of variability in the fMRI signal ( as we will observe later ) . Failing to account for sentence boundaries and working memory might be a source of confounding that has been ignored in the literature . Part-of-speech tags and dependency tags We use two standard word-level syntactic features - POS and DEP tags . The POS tag of a word is read off previously generated constituency trees ( those obtained using the Kitaev and Klein [ 32 ] parser ) . The DEP tag of a word ( e.g . subject , object , etc . ) corresponds to its assigned role in the dependency trees of the presented sentences which were generated using the spaCy English dependency parser [ 37 ] . We create one-hot binary vectors indicating the POS tag and the DEP tag of each word and concatenate them to create one feature space which we refer to as simple syntactic structure embeddings . Semantic features We adapt the vectors obtained from layer 12 of a pretrained [ 38 ] cased BERTlarge model [ 21 ] to identify regions that process semantics . We use layer 12 because of previous work showing that middle layers of sentence encoders are optimal for predicting brain activity [ 27 , 28 ] . To compute the semantic embedding of a word , we first feed the word and the words that precede it in its sentence to the pretrained model , preventing the inclusion of future semantic information . Then , we average the embeddings of all the input words and use this average to represent the semantic information seen till then . Using principal component analysis ( PCA ) , we reduce the dimensionality of the average to 15 to match the ConTreGE vectors ’ dimensionality and to also reduce overfitting ( the BERT embeddings have 1024 dimensions and we only have 1291 time points in the fMRI data ) . fMRI data We use the fMRI data of 9 subjects reading chapter 9 of Harry Potter and the Sorcerer ’ s Stone [ 39 ] , collected and made available freely without restrictions by Wehbe et al . [ 17 ] . Participants gave their written informed consent and the study was approved by the Carnegie Mellon University Institutional Review Board [ 17 ] . Words were presented serially for 0.5s each . All brain plots shown here are averages over the 9 subjects in the Montreal Neurological Institute ( MNI ) space . Further details are in Appendix A . Predicting brain activity The applicability of a given syntactic feature in studying syntactic processing is determined by its efficacy in predicting the brain data described above . Ridge regression is used to perform these predictions and their coefficient of determination ( R2 score ) measures the feature ’ s efficacy . For each voxel of each subject , the regularization parameter is chosen independently . We use ridge regression because of its computational efficiency and because of the Wehbe et al . [ 40 ] results showing that with such fMRI data , as long as the regularization parameter is chosen by cross-validation for each voxel independently , different regularization techniques lead to similar results . Indeed , ridge regression is a common regularization technique used for predictive fMRI models [ 41 , 17 , 16 ] . These models that predict brain activity are called voxelwise encoding models . For every voxel , a model is fit to predict the signals Y = [ y1 , y2 , . . . , yn ] recorded in that voxel where n is the number of time points ( TR , or time to repetition ) . The words are first grouped by the TR in which they were presented . Then , features of words in every group are summed to form a sequence of features X = [ x1 , x2 , . . . , xn ] aligned with the brain signals . The response measured by fMRI is an indirect consequence of neural activity peaking about 6 seconds after stimulus onset . A common way to account for this delay is to express brain activity as a function of the features of the preceding time points [ 41 , 17 , 16 ] . Thus , we train our models to predict any yi using xi−1 , xi−2 , xi−3 and xi−4 . We test the models in a cross-validation loop : the data is first split into 4 contiguous and equal sized folds . Each model uses three folds of the data for training and one fold for evaluation . We remove the data from the 5 TRs which either precede or follow the test fold from the training set of folds . This is done to avoid any unintentional data leaks since consecutive yis are correlated with each other because of the lag and continuous nature of the fMRI signal . The brain signals and the word features which comprise the training and testing data for each model are individually Z-scored . After training we obtain the predictions for the validation fold . The predictions for all folds are concatenated ( to form a prediction for the entire experiment in which each time point is predicted from a model trained without that time point ) . Note that since all 3 ConTreGE vectors are stochastic , we construct them 5 times each , and learn a different model each time . The 5 model predictions are averaged together into a single prediction . The R2 score is computed at each voxel using the predicted and real signals . We run a permutation test to test if R2 scores are significantly higher than chance . We permute blocks of contiguous fMRI TRs , instead of individual TRs , to account for the slowness of the underlying hemodynamic response . We choose a common value of 10 TRs [ 18 ] . The predictions are permuted within fold 5000 times , and the resulting R2 scores are used as an empirical distribution of chance performance , from which the p-value of the unpermuted performance is estimated . We also run a bootstrap test to test if a model has a higher R2 score than another . The difference is that in each iteration , we sample with replacement the predictions of both models for a block of TRs and compute the difference of their R2 and use the resulting distribution to estimate the p-value of the unpermuted difference . Finally , the Benjamni-Hochberg False Discovery Rate correction [ 42 ] is used for all tests ( appropriate because fMRI data is considered to have positive dependence [ 43 ] ) . The correction is performed by grouping together all the voxel-level p-values ( i.e . across all subjects and feature groups ) and choosing one threshold for all of our results . The correction is done in this way as we test multiple prediction models across multiple voxels and subjects . To compute Region of Interest ( ROI ) statistics , left-hemisphere ROI masks for the language system obtained from a “ sentence vs. non-word '' fMRI contrast [ 26 ] are obtained from [ 44 ] and mirrored to obtain right-hemisphere ROIs .

Summary: In this paper, the authors propose new representational spaces that capture syntactical information in natural language above and beyond complexity-based metrics that have been typically used in language neuroscience. First, they build representations based on the constituency tree of a sentence as every word is processed by the parser. Then, they use a subgraph embedding algorithm to convert the subtree into a 15-D representation. To test several hypothesis about the type of syntactic information being encoded, the authors look at 3 different types of subtrees- the largest subtree completed when the current word is processes, the incomplete subtrees that can explain the sequence seen thus far under PSG production rules and the set of complete parses produced by a probabilistic model that are weighted by their probabilities. The 3 subspaces are compared against 4 complexity/load-based metrics and additionally, a semantic feature space derived from BERT. They go on to build encoding models for the different feature spaces and compare them through variance partitioning. Overall, the paper finds that the 3 graph-based spaces explain additional variance across the cortex as compared to traditional complexity-based features. Further, this effect is not localized to any region. Instead, it seems to be distributed across a network that is largely explained by semantic features, suggesting that syntax and semantic share neural substrates and there exists no isolated region for syntactic processing.  The study uses publicly released fMRI data from 9 subjects reading a chapter from Harry Potter .