Syntactic representations in the human brain: beyond effort-based metrics

1 INTRODUCTION . Neuroscientists have long been interested in how the brain processes syntax . To date , there is no consensus on which brain regions are involved in processing it . Classically , only a small number of regions in the left hemisphere were thought to be involved in language processing . More recently , the language system was proposed to involve a set of brain regions spanning the left and right hemisphere ( Fedorenko & Thompson-Schill , 2014 ) . Similarly , some findings show that syntax is constrained to specific brain regions ( Grodzinsky & Friederici , 2006 ; Friederici , 2011 ) , while other findings show syntax is distributed throughout the language system ( Blank et al. , 2016 ; Fedorenko et al. , 2012 ; 2020 ) . The biological basis of syntax was first explored through studies of the impact of brain lesions on language comprehension or production ( Grodzinsky , 2000 ) and later through non-invasive neuroimaging experiments that record brain activity while subjects perform language tasks , using methods such as functional Magnetic Resonance Imaging ( fMRI ) or electroencephalography ( EEG ) . These experiments usually isolate syntactic processing by contrasting the activity between a difficult syntactic condition and an easier one and by identifying regions that increase in activity with syntactic effort ( Friederici , 2011 ) . An example of these conditions is reading a sentence with an object-relative clause ( e.g . “ The rat that the cat chased was tired '' ) , which is more taxing than reading a sentence with a subject-relative clause ( e.g . “ The cat that chased the rat was tired '' ) . In the past decade , this approach was extended to study syntactic processing in naturalistic settings such as when reading or listening to a story ( Brennan et al. , 2012 ; Hale et al. , 2018 ; Willems et al. , 2015 ) . Because such complex material is not organized into conditions , neuroscientists have instead devised effort-based metrics capturing the word-by-word evolving syntactic demands required to understand the material . Brain regions with activity correlated with those metrics are suggested to be involved in processing syntax . We use the term effort-based metrics to refer to uni-dimensional measures capturing word-by-word syntactic demands . A standard approach for constructing a syntactic effort-based metric is to assume a sentence ’ s syntactic representation and estimate the number of syntactic operations performed at each word . Node Count is popular such metric . It relies on constituency trees ( structures that capture the hierarchical grammatical relationship between the words in a sentence ) . While traversing the words of the sentence in order , subtrees of the constituency tree get completed ; Node Count refers to the number of such subtrees that get completed at each word , effectively capturing syntactic load or effort . Brennan et al . ( 2012 ) use Node Count to support the theory that the Anterior Temporal Lobe ( ATL ) is involved in syntactic processing . Another example of an effort-based metric is given by an EEG study by Hale et al . ( 2018 ) . They show that parser action count ( the number of possible actions a parser can take at each word ) is predictive of the P600 , a positive peak in the brain ’ s electrical activity occurring around 600ms after word onset . The P600 is hypothesized to be driven by syntactic processing ( to resolve incongruencies ) , and the results of Hale et al . ( 2018 ) align with this hypothesis . Though effort-based metrics are a good proposal for capturing the effort involved in integrating a word into the syntactic structure of a sentence , they are not reflective of the entire syntactic information in play . Hence , these metrics can not be used to study the brain representation of syntactic constructs such as nouns , verbs , relationships and dependencies between words , and the complex hierarchical structure of phrases and sentences . Constituency trees and dependency trees are the two main structures that capture a sentence ’ s syntactic structure . Constituency trees are derived using phrase structure grammars that encode valid phrase and clause structure ( see Figure 1 ( A ) for an example ) . Dependency trees encode relations between pairs of words such as subject-verb relationships . We use representations derived from both types of trees . We derive word level dependency ( DEP ) labels from dependency trees , and we focus on encoding the structural information given by constituency trees since we want to analyze if the brain builds hierarchical representations of phrase structure . We characterize the syntactic structure inherent in sentence constituency trees by computing an evolving vector representation of the syntactic structure processed at each word using the subgraph embedding algorithm by Adhikari et al . ( 2018 ) . We show that our syntactic structure embeddings – along with other simpler syntactic structure embeddings built using conventional syntactic features such as part-of-speech ( POS ) tags and DEP tags – are better than effort-based metrics at predicting the fMRI data of subjects reading text . This indicates that representations of syntax , and not just syntactic effort , can be observed in fMRI . We also address the important question of whether regions that are predicted by syntactic features are selective for syntax , meaning they are only responsive to syntax and not to other language properties such as semantics . To answer this question , we model the semantic properties of words using a contextual word embedding space ( Devlin et al. , 2018 ) . We find that regions that are predicted by syntactic features are also predicted by semantic features and thus are not selective for syntax . Scientific questions We ask three main questions : • How can scientists construct syntactic structure embeddings that capture the syntactic structure inherent in phrases and sentences ? • Are these embeddings better at predicting brain activity compared to effort-based metrics when used as inputs to encoding models ? • Which brain regions are involved in processing complex syntactic structure and are they different from regions involved in semantic processing ? Contributions We make four main contributions : • We propose a subgraph embeddings-based method to model the syntactic structure inherent in phrases and sentences . • We show that effort-based metrics can be complemented by syntactic structure embeddings which can predict brain activity to a larger extent than effort-based metrics . • Using our syntactic structure embeddings , we find some evidence supporting the hypothesis that the brain processes and represents complex syntactic information such as phrase and clause structure . • We find evidence supporting the existing hypothesis that syntactic processing appears to be distributed in the language network in regions that are not selective for syntax . 2 METHODS . We first describe the syntactic features used in this study and their generation . All of the features we use are incremental i.e . they are computed per word . We then describe our fMRI data analyses . Effort-based metrics We use four effort-based metrics in our analyses - Node Count , Syntactic Surprisal , word frequency and word length . Node Count is an effort-based metric popular in neuroscience . To compute it , we obtain the constituency tree of each sentence using the self-attentive encoder-based constituency parser by Kitaev & Klein ( 2018 ) . We compute Node Count for each word as the number of subtrees that are completed by incorporating this word into its sentence . Syntactic Surprisal is another effort-based metric proposed by Roark et al . ( 2009 ) and is computed using an incremental top down parser ( Roark , 2001 ) . Both of these metrics aim to measure the amount of effort that is required to integrate a word into the syntactic structure of its sentence . The word frequency metric is computed using the wordfreq package ( Speer et al. , 2018 ) as the Zipf frequency of a word . This is the base-10 logarithm of the number of occurrences per billion of a given word in a large text corpus . Finally , word length is the number of characters in the presented word . The last two metrics approximate the amount of effort that is required to read a word . Constituency tree-based Graph Embeddings ( ConTreGE ) Constituency trees are a rich source of syntactic information . We build three representations of these trees that encode this information : ( a ) The largest subtree which is completed upon incorporating a word into a sentence ( see figure 1 ( B ) ) is representative of the implicit syntactic information given by the word . Given that Node Count reduces all of the information present in these subtrees to just one number , it is easy to see that it can not effectively capture this information . POS tags ( categorize words into nouns , verbs , adjectives , etc . ) also capture some of the information present in these trees as they encode phrase structure to a certain extent . But , they are incapable of completely encoding their hierarchical structure and the parsing decisions which are made while generating them . In order to better encode their structure , we first build subgraph embeddings of these completed subtrees called ConTreGE Comp vectors . ( b ) We hypothesize that the brain not only processes structure seen thus far but also predicts future structure from structure it already knows . To test this , we construct embeddings , simply called ConTreGE vectors , using incomplete subtrees that are constructed by retaining all the phrase structure grammar productions that are required to derive the words seen till now , thereby allowing us to capture future sentence structure ( in the form of future constituents ) before the full sentence is read ( see figure 1 ( C ) ) . These subtrees contain leaves that are non-terminal symbols unlike complete subtrees that only have terminal symbols ( words and punctuation ) as leaves . In this context , a non-terminal symbol is a symbol that can be derived further using some rule in the phrase structure grammar ( ex . NP , VP , etc. ) . If incomplete subtrees are more representative of the brain ’ s processes , it would mean that the brain expects certain phrase structures even before the entire phrase or sentence is read . ConTreGE Comp and ConTreGE vectors need to be built using accurate constituency trees constructed using the whole sentence . Thus , we reuse the trees generated to compute Node Count to build them . ( c ) Further , the brain could be computing several possible top down partial parses that can derive the words seen thus far ( see figures 1 ( D ) and ( E ) ) and modifying the list of possible parses as future words are read . To test this hypothesis , we designed Incremental ConTreGE ( InConTreGE ) vectors that are representative of the most probable parses so far . For a given word , its InConTreGE vector is computed as : v = ∑5 i=1 e −siWi where Wi is the subgraph embedding of a partial parse tree built by an incremental top-down parser ( Roark 2001 CoLing ) after reading the word and si is the score assigned to this partial parse that is inversely proportional to the parser ’ s confidence in this tree . To effectively capture the structure of all subtrees , we encode them using the subgraph embeddings proposed by Adhikari et al . ( 2018 ) which preserve the neighbourhood properties of subgraphs . A long fixed length random walk on a subgraph is generated to compute its embedding . Since consecutive nodes in a random walk are neighbours , a long walk can effectively inform us about the neighbourhoods of nodes in the subgraph . Each node in a walk is identified using its unique ID . So , a random walk can be interpreted as a “ paragraph '' where the words are the node IDs . Finally , the subgraph ’ s embedding is computed as the Paragraph Vector ( Le & Mikolov , 2014 ) of this paragraph that is representative of the subgraph ’ s structure . Note that all of the subtrees of a given type ( complete , incomplete or partial parse ) are encoded together . This ensures that all ConTreGE Comp vectors , all ConTreGE vectors and all InConTreGE vectors are in our own spaces . Figure 2 illustrates the subtree encoding process . First , every unique non-terminal in the subtrees is mapped to a unique number ( ex . S is mapped to 1 , NP is mapped to 2 , etc . ) and every terminal is mapped to a unique number that is representative of the order in which they were presented ( the first presented token is mapped to 10000 , the second token is mapped to 10001 and so on ) . We did not map each unique terminal to a unique number ( for instance , we did not map all instances of `` Harry '' to one number ) because a random walk through the tree could give us word co-occurrence information and thus lead to the inclusion of some semantic information in the vectors . Every tree node ’ s label is then replaced by the number it was mapped to in the previous step . The edge lists of these subtrees are supplied to the subgraph embedding generation algorithm to finally obtain 15-dimensional vectors for every presented word . The length of the random walks is set to 100000 and we use an extension of the Distributed Bag of Nodes ( DBON ) model proposed by Le & Mikolov ( 2014 ) for generating Paragraph Vectors called Sub2Vec-DBON by Adhikari et al . ( 2018 ) . The length of the sliding window is set to 5 and the model is trained for 20 epochs . Since ConTreGE Comp , ConTreGE and InConTreGE encode information about the neighbourhoods of all nodes in the constituency trees , they can capture their hierarchical structure . Thus , brain regions predicted by these vectors are likely to be involved in building and encoding hierarchical sentence structure . Punctuation We create one-hot binary vectors indicating the type of punctuation that was presented along with a word ( e.g. . or , ) . For example , a sentence might have ended with `` Malfoy. '' . In this punctuation-based feature space , the column corresponding to . will be set to 1 for this word . While punctuation is seldom considered a syntactic feature , sentence boundaries are highly correlated with changes in working memory load . These changes are bound to be a great source of variability in the fMRI signal ( as we will observe later ) . Failing to account for sentence boundaries and working memory might be a source of confounding that has been ignored in the literature . Part-of-speech tags and dependency tags We use two standard word-level syntactic features - POS and DEP tags . The POS tag of a word is read off previously generated constituency trees . The DEP tag of a word ( ex . subject , object , etc . ) correspond to its assigned role in the dependency trees of the presented sentences which were generated using the spaCy English dependency parser ( 2 ) . We create one-hot binary vectors indicating the POS tag and the DEP tag of each word and concatenate them to create one feature space which we refer to as simple syntactic structure embeddings . Semantic features We adapt the vectors obtained from layer 12 of a pretrained ( 1 ) cased BERTlarge model ( Devlin et al. , 2018 ) to identify regions that process semantics . We use layer 12 because of previous work showing that middle layers of sentence encoders are optimal for predicting brain activity ( Jain & Huth , 2018 ; Toneva & Wehbe , 2019 ) . We obtain the contextual embeddings for a word by running the pretrained model only on the words seen thus far , preventing the inclusion of future semantic information . Since a presented word can be broken up into multiple subtokens , we compute its embedding as the average of the subtokens ’ embeddings . Using principal component analysis ( PCA ) , we reduce their dimensionality to 15 to match the ConTreGE vectors ’ dimensionality . fMRI data We use the fMRI data of 9 subjects reading chapter 9 of Harry Potter and the Sorcerer ’ s Stone ( Rowling , 2012 ) , collected and made available by Wehbe et al . ( 2014 ) . Words are presented one at a time at a rate of 0.5s each . All the brain plots shown here are averages over the 9 subjects in the Montreal Neurological Institute ( MNI ) space . Preprocessing details are in Appendix B . Predicting brain activity The applicability of a given syntactic feature in studying syntactic processing is determined by its efficacy in predicting the brain data described above . Ridge regression is used to perform these predictions and their coefficient of determination ( R2 score ) measures the feature ’ s efficacy . For each voxel of each subject , the regularization parameter is chosen independently . We use Ridge regression because of its computational efficiency and because of the Wehbe et al . ( 2015 ) results showing that with such fMRI data , as long as the regularization parameter is chosen by cross-validation for each voxel independently , different regularization techniques lead to similar results . Indeed , Ridge regression is a common regularization technique used for predictive fMRI models ( Mitchell et al. , 2008 ; Nishimoto et al. , 2011 ; Wehbe et al. , 2014 ; Huth et al. , 2016 ) . For every voxel , a model is fit to predict the signals Y = [ y1 , y2 , . . . , yn ] recorded in that voxel where n is the number of time points ( TR , or time to repetition ) . The words are first grouped by the TR in which they were presented . Then , the features of words in every group are summed to form a sequence of features X = [ x1 , x2 , . . . , xn ] aligned with the brain signals . The response measured by fMRI is an indirect consequence of brain activity that peaks about 6 seconds after stimulus onset . A common solution to account for this delay is to express brain activity as a function of the features of the preceding time points ( Nishimoto et al. , 2011 ; Wehbe et al. , 2014 ; Huth et al. , 2016 ) . Thus , we train our models to predict any yi using xi−1 , xi−2 , xi−3 and xi−4 . We test the models in a cross-validation loop : the data is first split into 4 contiguous and equal sized folds . Each model uses three folds of the data for training and one fold for evaluation . We remove the data from the 5 TRs which either precede or follow the test fold from the training set of folds . This is done to avoid any unintentional data leaks since consecutive yis are correlated with each other because of the lag and continuous nature of the fMRI signal . The brain signals and the word features which comprise the training and testing data for each model are individually Z-scored . After training we obtain the predictions for the validation fold . The predictions for all folds are concatenated ( to form a prediction for the entire experiment in which each time point is predicted from a model trained without the data for that time point ) . Note that since all 3 ConTreGe vectors are stochastic , we construct them 5 times each , and learn a different model each time . The predictions of the 5 models are averaged together into a single prediction . The R2 score is computed for every voxel using the predictions and the real signals . We run a permutation test to test if R2 scores are significantly higher than chance . We permute blocks of contiguous fMRI TRs , instead of individual TRs , to account for the slowness of the underlying hemodynamic response . We choose a common value of 10 TRs ( Deniz et al. , 2019 ) . The predictions are permuted within fold 5000 times , and the resulting R2 scores are used as an empirical distribution of chance performance , from which the p-value of the unpermuted performance is estimated . We also run a bootstrap test to test if a model has a higher R2 score than another . The difference is that in each iteration , we permute ( using the same indices ) the predictions of both models and compute the difference of their R2 and use the resulting distribution to estimate the p-value of the unpermuted difference . Finally , the Benjamni-Hochberg False Discovery Rate correction ( Benjamini & Hochberg , 1995 ) is used for all tests ( appropriate because fMRI data is considered to have positive dependence ( Genovese , 2000 ) ) . The correction is performed by grouping together all the voxel-level p-values ( i.e . across all subjects and feature groups ) and choosing one threshold for all of our results . The correction is done in this way since we test multiple prediction models across multiple voxels and subjects . To compute Region of Interest ( ROI ) statistics , left-hemisphere ROI masks for the language system obtained from a “ sentence vs. non-word '' fMRI contrast ( Fedorenko et al. , 2010 ) are obtained from ( 3 ) and mirrored to obtain the right-hemisphere ROIs .

Summary: This paper derives various types of graph embeddings to encode aspects of syntactic information that the brain may be processing during real-time sentence comprehension. These embeddings, along with indicators of punctuation, POS and dependency tags, and BERT embeddings, are used to predict brain activity recorded via fMRI. The authors argue that this is an improvement over use of effort-based metrics to predict brain activity, as these embeddings contain richer information than is captured by distilling down to a single measure of effort. They show that various brain regions are significantly better predicted by the syntactic embeddings than by the effort-based metrics and POS+dependency indicators. BERT embeddings, however, prove to be a better predictor (than syntactic and other predictors) across much more substantial areas of activity.