Can fMRI reveal the representation of syntactic structure in the brain?
Authors: Aniketh Janardhan Reddy
Leila Wehbe
Venue: NeurIPS 2021 Poster (2021)
Keywords: neuroscience
fMRI
syntactic representations
graph embeddings

Abstract: While studying semantics in the brain, neuroscientists use two approaches. One is to identify areas that are correlated with semantic processing load. Another is to find areas that are predicted by the semantic representation of the stimulus words. However, most studies of syntax have focused only on identifying areas correlated with syntactic processing load.  One possible reason for this discrepancy is that representing syntactic structure in an embedding space such that it can be used to model brain activity is a non-trivial computational problem. Another possible reason is that it is unclear if the low signal-to-noise ratio of neuroimaging tools such as functional Magnetic Resonance Imaging (fMRI) can allow us to reveal the correlates of complex (and perhaps subtle) syntactic representations. In this study, we propose novel multi-dimensional features that encode information about the syntactic structure of sentences. Using these features and fMRI recordings of participants reading a natural text, we model the brain representation of syntax. First, we find that our syntactic structure-based features explain additional variance in the brain activity of various parts of the language system, even after controlling for complexity metrics that capture processing load. At the same time, we see that regions well-predicted by syntactic features are distributed in the language system and are not distinguishable from those processing semantics. Our code and data will be available at https://github.com/anikethjr/brain_syntactic_representations.

Can fMRI reveal the representation of syntactic structure in the brain?
Aniketh Janardhan Reddy Machine Learning Department Carnegie Mellon University a ajreddy@cs cmu. edu
Leila Wehbe Machine Learning Department Carnegie Mellon University lwehbe@cmu. edu
Abstract
While studying semantics in the brain, neuroscientists use two approaches. One is to identify areas that are correlated with semantic processing load. Another is to find areas that are predicted by the semantic representation of the stimulus words. However, most studies of syntax have focused only identifying areas correlated with syntactic processing load. One possible reason for this discrepancy is that representing syntactic structure in embedding space such that it be used to model brain activity a non-trivial computational problem. Another possible reason is that it is unclear if the low aggaal-t-noise ratio of neuroimaging tools such as functional Magnetic Resonance Imaging (fMRI) can allow us to reveal the correlates of complex (and perhaps subtle) syntactic representations. In this study, we propose novel multi-dimensional features that encode information about the syntactic structure of sentences. Using these features and fMRI recordings of participants reading a natural text, we model the brain representation of syntax. First, we find that our syntactic structure-based features explain additional variance in the brain activity of various parts of the language system, even after controlling for complexity metrics that capture processing load. At the same time, we see that regions well- eell-predicted by syntactic features are distributed in the language system and are distinguishable from those processing semantics. Our code and data will be available at https://gihuub.oo/mmmieeee htt.eeehh//bbaaasstttttiiiiiiic syntactii_reppiiitcee representations.
1 Introduction
Neuroscientists have long been interested in how the brain processes syntax. To date, there is no consensus which brain regions involved in processing it. Classically, only small number of regions the left hemisphere were thought to be involved in language processing. More recently, the language system was proposed involve set of brain regions spanning the left and right hemisphere [] Similarly, some findings show that syntax is constrained to specific brain regions 22 3], while other findings show syntax is distributed throughout the language system 4.
The biological basis of syntax was first explored through studies of the impact of brain lesions on language comprehension production [7] and later through non non-invasive neuroimaging experiments that record brain activity while subjects perform language tasks, using methods such as functional Magnetic Resonance Imaging (fMRI) or electroencephaloggrphyy (EEG). These experiments usually isolate syntactic processing by contrasting the activity between difficult syntactic condition and an easier one and by identifying regions that increase in activity with syntactic effort [3]. An example of these conditions is reading a sentence with an object-relative clause (e.g. "The rat that the cat chased was tired"), which more taxing than reading sentence with a subject-relative clause (e.g. "The cat that chased the rat was tired"). In the past decade, this approach was extended to study syntactic processing in naturalistic settings such when reading or listening a story (8- Because such complex material is not organized into conditions, neuroscientists have instead devised
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

complexity metries capturing the word-by-word evolving syntactic demands required to understand the material. Using these metrics, neuroscientists were able to identify regions with activity correlated with syntactic processing load, and suggest the involvement of these regions in syntactic processing.
While many have studied syntactic processing as captured through complexity measures, very few have studied the syntactic representations themselves. As an analogy, consider the neurobiology of language semantics. A large part of the literature has focused on characterizing the semantic processing load related to integrating or predicting incoming words (see [2] for review). Concur- rently, another part has focused on studying where the meaning of words itself is represented, either through contrast- based studies (see [I3] for a review) or through encoding model approaches (e.g. [1)19). It can be argued however, that in syntax, most research focuses on the first approach (using complexity) and not the second (using representations).
There are two main reasons why the study of syntactic representations using fMRI is difficult:
The first reason is computational. To identify the brain correlates of syntactic representations, one has to embed the syntactic representation (often a tree) into vector that can be then used to predict the time series brain activity as a function of syntactic structure. This vector representation should change as the words are processed incrementally. The construction of such a vector space is akin to the problem of building a graph embedding, and it is not trivial. The goal is to have different sentences segments of sentences with similar structure (irrespective of their meanings) map to nearby points in the vector space.
2. The second reason is that the fMRI signal is noisy, and it is not clear that the neural basis of the representation of syntactic structure lends itself being studied using fMRI. For instance, it could be that the neural substrate for syntax is intermingled with that of other language components like semantics and is hard to disentangle, or it could be that neurons inside the same voxel perform different syntactic computations, making it hard to differentiate the signal corresponding to these different computations. More generally, it could also be that syntactic computations are organized in Ratio (SNR) of fMRI makes it difficult to study them.
way that the low Signal-to-Nor
To help address the first difficulty, we propose syntactic structure embeddings that encode the syntactic information inherent in natural text that subjects read in the scanner. These structure embeddings are proposed as an additional tool in the arsenal of neurolinguists and can serve to ask the question of whether the fMRI signal can reveal syntactic representations. We use our syntactic structure embeddings in voxelwise encoding model framework [4 16 17 20). We find that our embeddings along with other simpler syntactic structure embeddings built using conventional syntactic features such as part-of-speech (POS) tags and dependency role (DEP) tags are able to explain an additional portion of the variance in the fMRI data of subjects reading text, even after controlling for complexity metrics that capture processing load, suggesting that fMRI can indeed reveal syntactic representations. Furthermore, the regions that are well predicted by syntactic representations are distributed across the language network.
We also attempt to minimize the amount of semantic information present in our embeddings. After showing that they do not encode a significant amount of semantics, we use our syntactic embeddings to address the aforementioned issue of whether regions that are predicted by syntactic features are selective for syntax, meaning they are only responsive to syntax and not to other language properties such as semantics. To answer this question, we use a contextual word embedding space (21 that integrates semantics and syntax. Consistent with prior literature, regions that are predicted by syntax much better predicted by the contextual embeddings and do not appear to be selective for syntax.
2 Background
Naturalistic tasks are gradually becoming the paradigm of choice to study language processing in the brain. Many have used naturalistic listening tasks investigate syntactic processing by measuring word-by-wor by-rrd processing load (8 10 22 25). Even though story reading or listening involves multiple types of processing (low-level processing, word recognition, building sentences, inferring meaning and social reasoning, etc.), researchers are gradually learning more and more about how these processes are mapped to different brain areas. It has already been shown that the regions defined by Fedorenko et al. (26) as the language network (also used in this work)
Naturalistic tasks
process word-level syntactic [16-18] and semantic features as well as sequence meaning [1 27 28
Processing information beyond the level of sentences is thought to recruit other areas 17 29

---

Encoding models are frequently used to analyze data collected using naturalistic tasks. These are models that are trained to predict brain activity at every voxel as function of some feature of interest. Voxels that are well-predicted are thought to be involved in processing the information encoded in the feature. The various complexity metries were formulated as features that encode syntactic demands.
Complexity metrics Several complexity metries have been proposed to capture word-by-word syntactic demands. One common approach for constructing syntactic complexity metric is to assume a sentence's 's syntactic representation and estimate the number of syntactic operations performed at each word. Node Count is a popular metric constructed using this approach (30. It relies on constituency trees (structures that capture the hierarchical grammatical relationship between the words in sentence). While traversing the words of the sentence in order, subtrees of this tree get completed; Node Count refers to the number of such subtrees that get completed at each word, effectively capturing syntactic load or effort. Important work in neurolinguistics support the theory that the Anterior Temporal Lobe (ATL.) is involved in syntactic processing.
[8] has used Node Count to
Complexity metries are good proposal for identifying brain regions involved in processing the syntactic structure of a sentence. They can be used to study specific hypotheses about how syntactic structure is built. For example, one could compute these metries using different types of parsers to determine which parser's ss operations are more similar to the brain's 's structure building processes [22] Using these metrics also leads more robust results when compared to those obtained by contrasting syntactic conditions. This is because of two reasons. First, the naturalistic stimuli used in such studies are very diverse. Second, since the metries can be computed at every word (i.e. they are incremental metrics), the correlation between syntactic processing load and the recorded brain activity can be robustly estimated using large number of data points (i.e. words).
Representing syntactic structure The focus of this work is to study syntactic representations in the brain by building features that encode a sentence's 's syntactic structure and using them in voxelwise encoding models. Constituency trees and dependency trees are the two main structures that capture a sentence's syntactic structure. Constituency trees are derived using phrase structure grammars (PSGs) that encode valid phrase and clause structure (see fig. T[A for an example). Dependency trees encode relations between pairs of words such as subject-vers relationships. We derive representations from both types of trees. We derive word-level DEP tags from dependency trees, and encode the structural information given by constituency trees to capture the hierarchical representations of phrase structure. We characterize the syntactic structure inherent in sentence constituency trees by computing an evolving vector representation of the syntactic structure processed at each word using the subgraph embedding algorithm by Adhikari et al. (31]
We resort to developing this new embedding pipeline because we want our constituency tree-based embeddings to be almost purely syntactic. One might suggest using the internal representations of a neural parser (such as the parser used by Hale et al. (9) to directly encode syntax. However, neural parsers use word embeddings. making it likely for there to be some semantics in the internal representations. This "leakage" of semantics makes it very difficult isolate the brain regions that process syntax. The problem is further compounded by previous findings which indicate that syntax and semantics might be processed in the same regions of the brain ( making it impossible to infer whether region's 's activity is correlated with the syntax or the semantics encoded in the features.
3 Methods
We first describe the syntactic features used in this study and their generation. All of the features we use are incremental i.e. they are computed per word, We then describe our fMRI data analyses.
Complexity metries We use four complexity metrics in our analyses Node Count, Syntactic Surprisal, word frequency and word length. To compute Node Count, we obtain the constituency tree of each sentence using the self-attentive encoder-based constituency parser by Kitaev and Klein [32] We compute Node Count for each word as the number of subtrees that are completed by incorporating this word into its sentence. Syntactic Surprisal another complexity metric proposed by Roark et al. 33 and is computed using an incremental top-down parser
3 It measures how unexpected it
to read a given word in the current syntactic context. Both of these metrics aim to measure the amount of effort that is required to integrate a word into the syntactic structure of its sentence. The word frequency metric is computed using the wordfreq package (35 as the Zipf frequency of a word. This is the base- 10 logarithm of the number of occurrences per billion of given word in a large text
3

---

corpus. Finally, word length is the number of characters in the presented word. The last two metrics approximate the amount of effort that required to read word, irrespective of its context.
NP
VP
NP
PP
NP
NP
MNP
VBD
VBD
VBD
IN
NN
Harry|
was
lown
with
her
flving
skilss
Htrrry
love
with
her
(yying
dillss
Harry
way
In
love
Harry
was
in
love
Figure Example of complete and incomplete subtrees and two possible partial parses: Part A shows a sentence's s constituency tree generated by a self-attentive encoder-based constitueccy parser 32 using all of its words. The largest completed subtree for "love" is highlighted in red. The incomplete subtree generated till "love" is highlighted in blue in part B. Incomplete subtrees are generally much deeper than complete ones. In part C, we can see two possible partial parses generated by an incremental top-down parser (34) only using the words till "love". We see that the structure of the two parses is slightly different. The supplementary material contains a document illustrating complete and incomplete subtrees for words from a long sentence.
Constituency Tree-based Graph Embeddings (ConTreGE)) Constituency trees are a rich source of syntactic information. We build three representations of these trees that encode this information:
1. The largest subtree which is completed upon incorporating a word into a sentence (see fig. [A) is representative of the implicit syntactic information given by the word. A subtree is considered complete when all of its leaves are terminals. The largest subtree completed by a given word refers to the subtree with the largest height that also satisfies the following conditions the given word must be one of its leaves and all of its leaves must only contain words that have been seen till then. Given that Node Count reduces all of the information present in these subtrees just one number, it is easy to see that it cannot fully capture this information. POS tags (categorize words into nouns, verbs, adjectives, etc.) also capture some of the information present in these trees as they encode lexical syntactic information. But, they are incapable of completely encoding their hicrarchical structure and the parsing decisions made while generating them. To better encode their structure, we first build subgraph embeddings of these completed subtrees called ConTreGE Comp vectors.
2. We hypothesize that the brain not only processes structure seen thus far but also predicts future structure from structure it already knows. To test this, we construct embeddings, called E Incomp vectors, using incomplete subtrees that are constructed by retaining all the PSG productions that are required to derive the words seen till then, starting from the root of the sentence's 's tree. A non-terminal node is expanded only if it eventually leads to the derivation of a word that has been seen. Any other non-terminal nodes are not expanded. Encoding such subtrees allows us to capture future sentence structure (in the form of future constituents) before the full sentence read (see fig. [B). These subtrees contain leaves that are non-terminal symbols unlike complete subtrees that only have terminal symbols (words and punctuation) as leaves. In this context, a non-telminal symbol is a symbol that can be derived further using some rule in the PSG (c.g. NP, VP, etc.). If incomplete subtrees are more representative of the brain's processes, it would mean that the brain correctly predicts certain phrase structures even before the entire phrase or sentence is read. ConTreGE Comp and ConTreGE Incomp vectors need to be built using accurate constituency trees constructed using the whole sentence. Thus, we reuse the trees generated to compute Node Count to build them.
3. Further, the brain could be computing several possible top-down partial parses that can derive the words seen thus far (see fig. [[)) and modifying the list of possible parses as future words are read. To test this hypothesis, we designed Incremental ConTreGE (InConTreGE) vectors that are representative of the most probable parses so far. For a given word, its InCon vector is computed as: v = Db -=1 4: W, where W; is the subgraph embedding of partial parse tree built by an incremental top-down parser
34 after reading the word and 8; is the score assigned to this partial parse that is inversely proportional to the parser's 's confidence in this tree. The InCon' TreGE feature space is constructed to encode the different possible parse trees that can derive the words seen so far. These different parse trees try to encode uncertainty about the parent nodes of the words seen so

---

1,221,
2.1.4,4.344
2.2.4, 52000,
0.251, 5.692
9.635, 7.53
6.55, 0.199
10003, 1
20003, 6
8.290
Generate long random walk the subtree
Computte Paragraph Vectors of the walk
Herry
loow
10003
b0002
30023
Convert symbols
Subtree encoded
to numbers
Figure 2: Steps for encoding subtrees.
far. They also do not encode any information about possible future children of these parent nodes. However, with ConTreGE Incomp, we encode parts the final parse tree sentence. It does not encode any uncertainty. With this embedding space to correctly predict future constituents. We see uncertainty about the current parse and the ability to correctly predict future constituents use the T Incomp to hypothesize about the brain encoding correct future information.
trying measure the ability of the brain
related but distinct concepts. This is the reason why
To effectively capture the structure of all subtrees, we encode them using the subgraph embeddings proposed by Adhikari al. [31] which preserve the neighbourhood properties of subgraphs. A long fixed length random walk on subgraph is generated to compute its embedding. Since consecutive nodes a random walk are neighbours, a long walk can inform us about the neighbourhoods of nodes in the subgraph. Each node in walk is identified using its unique ID. So. random walk can be interpreted as a "paragraph" where the words are the node IDs. Finally, the subgraph's 's embedding is computed the Paragraph Vector [36] of this paragraph that is representative of the subgraph structure. Note that all of the subtrees of a given type (complete, incomplete or partial parse) are encoded together, ensuring that all ConTreeGE Comp, all ConTreGE Incomp and all InConTreGE vectors are in their own spaces. Fig. [ iillustrates the subtree encoding process. First, every unique non-terminal in the subtrees mapped etc.) and every terminal is mapped to a unique number that is representative of the order in which they were presented (the first presented token is mapped 10001 and so on). We did not map each unique terminal to a unique number (for instance, we did not map all instances of "Harry" one number) because random walk through the tree could give us word co-occurrence information and thus lead to the inclusion of some semantics in the vectors.
unique number (e.g. S is mapped 1, NP mapped
10000, the second token is mapped to
Every tree node's label is then replaced by the number it was mapped to in the previous step. The edge lists of these subtrees are supplied to the subgraph embedding generation algorithm to finally obtain d5-dimensional vectors for every presented word. The length of the random walks is set to 100000 and we use an extension of the Distributed Bag of Words (DBOW) model proposed by Le and Mikolov [36] for generating Paragraph Vectors called
by Adhikari et al. (31). The sliding window length is set to 5 and the model is trained for 20 epochs. Since ConTreGE Comp, Con TreGE Incomp and MiConnreeG. encode information about the neighbourhoods of all nodes in the constituency trees, they can capture their hierarchical structure. Thus, brain regions predicted by these vectors are likely to be involved in building and encoding hierarchical sentence structure.
Punctuation We create one-hot binary vectors indicating the type of punctuation that was presented along with a word (e.g. or,). For example, sentence might have ended with "Malfoy.". ". In this punctuation-Be based feature space, the column corresponding to will be set to for this word. While punctuation is seldom considered syntactic feature, sentence boundaries are highly correlated with changes in working memory load. These changes are bound to be a great source of variability in the fMRI signal (as will observe later). Failing account for sentence boundaries and working memory might be a source of confounding that has been ignored in the literature.
Part-of-peeech tags and dependency tags We use two standard word-level syntactic features POS and DEP tags. The POS tag of a word is read off previously generated constituency trees (those obtained using the Kitaev and Klein [32 parser). The DEP tag of a word (e.g. subject, object, etc.) corresponds to its assigned role in the dependency trees the presented sentences which generated using the spaCy English dependency parser (37]. We create one-hot binary vectors indicating the POS tag and the DEP tag of each word and concatenate them to create one feature space which we refer as simple syntactic structure embeddings.
Semantic features We adapt the vectors obtained from layer 12 of pretrained cased BERT- large model [21] to identify regions that process semantics. We use layer 12 because of previous work
5

---

showing that middle layers of sentence encoders are optimal for predicting brain activity [27 28]. [8]. To
compute the semantic embedding of a word, we first feed the word and the words that precede it in its sentence to the pretrained model, preventing the inclusion of future semantic information. Then, we average the embeddings of all the input words and use this average to represent the semantic information seen till then. Using principal component analysis (PCA), we reduce the dimensionality of the average to 15 to match the ConTreGE vectors' dimensionality and to also reduce overfitting (the BERT embeddings have 1024 dimensions and we only have 1291 time points in the IMRI data).
IMRI data We use the fMRI data 9 subjects reading chapter of Harry Potter and the Sorcerer's 's Stone [39], collected and made available freely without restrictions by Wehbe et al. [17]. Participants gave their written informed consent and the study was approved by the Carnegie Mellon University Institutional Review Board [17]. Words were presented serially for 0.5s each. All brain plots shown here are averages over the subjects in the Montreal Neurological Institute (MNI) space. Further details are in Appendix A
Predicting brain activity The applicability of given syntactic feature in studying syntactic processing is determined by its efficacy in predicting the brain data described above. Ridge regression used perform these predictions and their coefficient of determination (/t² score) measures the feature's ' efficacy. For each voxel of each subject, the regularization parameter is chosen independently. We use ridge regression because of its computational efficiency and because of the Wehbe et al. [40] results showing that with such fMRI data, as long as the regularization parameter is chosen by cross-validation for each voxel independently, different regularization techniques lead similar results. Indeed, ridge regression is common regularization technique used for predictive fMRI models 17 16] These models that predict brain activity are called voxelwise encoding models.
For every voxel, a model is fit to predict the signals Y = [y11 92,..., ynl recorded in that voxel where is the number of time points (TR, or time to repetition). The words are first grouped by the TR in which they were presented. Then, features of words in every group are summed to form a sequence of features X (2,, ... , [nn]]] aligned with the brain signals. The response measured by fMRI an indirect consequence of neural activity peaking about 6 seconds after stimulus onset. A common way to account for this delay is to express brain activity as a function of the features of the preceding time points [1 17 16]. Thus, we train our models to predict any Y using 24-1, T4-2, 24-3 and 24-1-
We test the models in cross-validation loop: the data is first split into 4 contiguous and equal sized folds. Each model uses three folds of the data for training and one fold for evaluation. We remove the data from the 5 TRs which either precede This is done to avoid any unintentional data leaks since consecutive y:S are correlated with each other because of the lag and continuous nature of the fMRI signal. The brain signals and the word features which comprise the training and testing data for each model are individually Z-scored. After training we obtain the predictions for the validation fold. The predictions for all folds are concatenated (to form a prediction for the entire experiment in which each time point is predicted from model trained without that time point). Note that since all 3 ConTreGE vectors are stochastic, we construct them times each, and learn different model each time. The 5 model predictions are averaged together into a single prediction. The R² score is computed at each voxel using the predicted and real signals.
follow the test fold from the training set of folds.
We run a permutation test to test if R² scores are significantly higher than chance. We permute blocks of contiguous fMRI TRs, instead of individual TRs, account for the slowness of the underlying hemodynamic response. We choose a common value of 10 TRs [8) The predictions are permuted within fold 5000 times, and the resulting R2 scores are used as an empirical distribution of chance performance, from which the p-value of the unpermuted performance is estimated. We also run bootstrap test to test if a model has a higher R² score than another. The difference is that in each iteration, we sample with replacement the predictions of both models for block of TRs and compute the difference of their R² and use the resulting distribution to estimate the p-value of the unpermuted difference. Finally, the Denjamni-Hochler (appropriate because fMRI data considered to have positive dependence 43 The correction is performed by grouping together all the voxel-level p-values (i.e. across all subjects and feature groups) and choosing one threshold for all of our results. The correction is done in this way as we test multiple prediction models across multiple voxels and subjects. To compute Region of Interest (ROI) statistics, hefispheiee ROI masks for the language system obtained from a "sentence vs. non-word" fMRI contrast [26) are obtained from 44 and mirrored to obtain right- hegtsheeme ROIs.
False Discovery Rate correction 42 is used for all tests

---

Results
We use different sets of feature spaces as inputs to encoding models that predict activity in each fMRI voxel. Many of our feature spaces have overlapping information POS and DEP tags include punctuation, BERT vectors have been shown to encode syntax [45] and ConTreGE vectors, built from constituency trees, encode some POS tags information. To detect brain regions sensitive the distinct information given by a feature space, we build hierarchical feature groups in increasing order of syntactic information and test for significant differences prediction performance between two consecutive groups. We start with the simplest feature punctuation, and then add more complex features order: the complexity metries, POS and DEP tags, one of the ConTreGE vectors and the vectors derived from BERT (which can be thought of as a super-set of semantics and syntax). At each step, we test if the introduction of the new feature space leads to significantly larger than chance has the raw
improvement in R (similar to [46]. Fig. 3nd summarize our results (Appendix ( prediction results and Appendix D contains detailed plots for every ROI showing the subject-specine values).
(a) (PU)
(b) (U)
(c) (PD. CM. PUJ -ICM, PU)
(d) PD, CM. PU) ((PD, CM. PU)
1
4
5+
(C) PD. CM. PU) -(PD, CM, PU)
(DIINC, PD. CM. PU) -(PD, CM, PU)
(EETTT CLPD.CM.PU) -ICI, PD. CM, PU)
Figure 3: The first plot shows the number of subjects for which given voxel is significantly predicted by punctuation (p S 0.05). The others show the number of subjects for which the difference R scores between two feature groups is significant complexity metrics, PD = POS and DEP Tags, CC= Con Comp, CI= ConTreeG Incomp, INC InConTTreEE, BERT BERT embeddings and '(.)' indicates that these features were concatenated in order to make the predictions. the two feature groups being larger than O. We see that the distinct information given by syntactic structure-bases features does explain additional variance in the brain activity, even after controlling for the complexity metrics and punctuation. The semantic vectors are also very predictive and many well-predicted regions overlap with those that are predicted by syntax.
0.05). Here, PU Punctuation, CM All
indicates hypothesis test for the difference in R2 scores between
(CM, PU) - (PU)
(CL, PO. CM, PUJ (PD. CM, PU)
(INC, PD. PU)
PU)
[CC, CM. PU) (PO, PU)
(BERT. CL, PO. CM, PU) (CI, CM. FU)
-4
sexcl with significant
30
2
Anterior Temporil LobetAT))
B AnguAr Synsi66) Elteeoo Frontal Synsa (P3)
MMssse Tremral Dynas (MFfC)
Synus
PTL A01
A6
RDI
IFG
MFG
iFobbb
Figure 4: Region of Interest (ROI) analysis of the prediction performance. [Left] Language system ROIS by (26 from (4). [Right] For each model, the percentage of ROI voxels in which we see significant increases in prediction performance. Each bar represents the average percentage across subjects and the error bars show the standard error across subjects. We use the same abbreviations as in fig. 3 and see the same trends across ROIs.

---

Feature
Level 2
Level
Level 4
Level 5
Level
Level 7
Level 8
Level 9
Most Popular Label
51
38.76
54.42
64.05
73.44
78.25
82.38
85.82
All Complexity Metries
49.84
45.29*
59.66
67.99*
77.3*
82.13*
86.34*
89.74
POS and DEP tags
92.23*
71,27*
67.06
70"
77.51*
82.09*
86.26*
89,59*
ConTreGE Comp
66.76*
51,01*
59.52
67.95*
77.28
82.08*
86.26*
89.68*
ConTreGE Incomp
52.91
45,42*
57.37
67.29
76.87
82.3
86.51
90.39*
InConTreGE
52.46
45.59*
57.61
66.75
76.13
81.08
85.27*
89.1*
BERT Embeddings
52.18
45.73*
58.62
66.79
75.68
80.31
84.72
88.72
Table I: 10-fold cross validation accuracies in predicting the ancestors a word. denotes accuracies
significantly above chance (tested using Wilcoxon signed-rank test, p S 0.01). POS and DEP tags
best predict lower level ancestors while ConTreGE Incomp vectors best predict higher level ones.
Syntactic structure embeddings capture additional variance in the brain activity even after controlling for complexity metries From fig. (]b),, it can be seen that the information provided by the complexity metrics is predictive of brain activity to certain degree even after controlling for punctuation. Fig. 3 cf- show that when the features that encode syntactic structure are added to the feature groups, they explain even more variance, showing that fMRI can indeed reveal syntactic representations. These results are made even clearer by fig. The R scores in many voxels increase significantly (above what is predicted by the complexity metrics) after including POS and DEP tags and ConTreGE Incomp. ConTreGE Comp is not as predictive as Con Incomp. Additionally, InConTreGE is not as predictive as ConTreGE Incomp, suggesting that the top-down parser might be generating partial parses that are not reflective of brain representations.
Complex syntactic information is distributed in the brain In this section, we analyze the information in Con E Incomp to interpret its brain prediction performance. We estimate how much of the constituency tree captured by each feature by using to predict the level N ancestor of a word (in its constituency tree). We vary N from 2 to 9 and train a logistic regression model for each N. Since POS tags are the level ancestors of words, we start the analysis at N=2. As there are many phrase labels, we group them into 7 buckets noun phrases, verb phrases, adverb phrases, adjective phrases, prepositional phrases, clauses and other miscellaneous labels. If a word's depth in its tree is less than N, the root is considered its level N ancestor.
Table D shows the results of this analysis. We use the trees generated by the Kitaev and Klein 32 parser. Given the skewed label distribution, the optimal strategy for a predictor that takes random noise as input is always output the majority class ancestor at that level. Chance performance is thus equal to the frequency of the majority label. POS and DEP tags are predictive of labels at all levels and produce the highest accuracies for lower levels. The InConTreGE vectors are not as Incomp or Con Comp, hinting that the top-down parser might not be
predictive as very accurate. ConTreGE Incomp is the best predictor of higher level ancestors but ConTreGE Comp is better than Con Incomp at predicting lower level ancestors. This may be because graph embeddings of tree tend to capture more of the information near the tree's root (a random walk through a somewhat balanced tree is likely to contain more occurrences of nodes near the root and random walks are encoded in the subgraph embedding generation process). ConTreGE Comp vectors, created from shallow complete trees, likely over-represent lower level ancestors while ConTreGE Incomp vectors, created from relatively deeper trees, likely over-represent higher level ancestors. Given that ConTreGE Incomp contains information about the higher level ancestors of a word, this suggests that we are able to pick up on the distributed brain representations of complex hierarchical syntactic information such as phrase and clause structure, using this feature.
The ConTreGE Incomp vectors also encode future syntactic constituents. Thus, it is possible that this information is helpful in predicting the current brain activity. Although this result supports the hypothesis that the brain predicts future syntactic structure, we cannot conclusively prove it. This is because certain other characteristics of the feature space such as its smoothness might also make it more predictive. We leave it to future work to explore this hypothesis in more depth.
Syntax and semantics are processed in a distributed way in overlapping regions across the language system We first empirically show that our ConTreGE vectors do not encode a significant amount of semantics. We run a simple analysis similar to the one used by Caucheteux et al. [47] Like in Caucheteux et al. [47, we try to predict a Glove-based semantic vector (48 (extracted using spaCy 37 using the BERT embeddings and the Con vectors. These predictions were
performed using a simple ridge regression model whose regularization parameters were chosen based on leave-one-out cross- noss-validation R²2 scores (performed using sklearn's 's RidgeCV modulc). In order
8

---

to get the final R scores, we perform 10-fold vols-vvaliationn with separate RidgeCV model being fit for each train-test split. We find that the BERT embeddings achieve average R² scores of 0.185 (the R2 scores are first averaged across all 300 dimensions of the Glove-based semantic vectors and then across all 10 cross-validation splits). However, the graph embeddings achieve significantly lower scores with ConTreGEE Comp, ConTrreGE Incomp and InConTreCHt achieving average R² scores of 0.052, 0,020 and 0.021 respectively. These scores are markedly lower than the ones obtained using BERT, illustrating that the graph embeddings contain very low amounts of semantic information. On the other hand, our syntactic information analysis (Table I shows that these embeddings capture a lot of syntactic information. All of our other syntactic feature spaces are purely syntactic by definition.
Since the synactic feature spaces encode a lot of syntactic information. is reasonable to assume that any additional variance predicted by the BERT embeddings after controlling for the syntactic feature spaces, at least partially due to their semantic information. From Figures 3 and 4 BERT embeddings explain additional variance in most of the regions in the language system and these regions overlap with the regions that are predicted by the syntactic feature spaces. As we have already established that the syntactic feature spaces contain a small amount of semantics, we can thus conclude that syntactic and semantic information are processed in a distributed fashion across the language network and that there are no regions that are selective for syntax.
we see that the
5 Discussion and Related Work
Syntactic representations Multiple complexity metrics have been proposed. Hale al. [9] use parser action count (the number of possible actions a parser can take at each word) to model EEG data. They find that it is predictive of the P600, a positive peak in the brain's 's electrical activity occurring around 600ms after word onset. The P600 is hypothesized to be driven by syntactic processing (to resolve incongruencies), and the results of Hale et al. 9 align with this hypothesis. Brennan et al. [22] derive the surprisal and node count metrics using nine different models (metrics are computed only the model allows their computation, e.g. Node Count cannot be computed using Markov models). Each model captures syntactic information at one of three levels of abstraction most basic word-to-word dependencies (Markov models), hierarchy of phrases (context-free PSG) and the most abstract level which captures long-distame dependencies (Minimalist Grammars [49 50)). They observe that the metrics computed using the hicrarchical grammars are predictive of fMRI signals from the left ATL and PTL which were collected during natural listening. The metrics derived from the Markov models predict the same regions along with the IFG.
use complexity metrics to study syntactic processing during natural reading or listening. However, a few studies do explicitly encode syntactic structure: Wehbe et al. [17] find that POS and DEP tags are the most predictive out of set of word, sentence and discourse-level features. Concurrent with our work, Caucheteux et al. [47] also try to build syntactic representations using GPT-2 [52) embeddings and obtain results similar to ours. Complementing popular approaches that are dependent on complexity metrics, we built upon the work of Wehbe et al. [17] by developing novel graph embeddings-ben based approach to explicitly capture the syntactic information provided by constituency trees. Our results show that these explicit features have more syntactic information than complexity metries and this information is able to predict additional variance in the brain activity. Given these results, we believe that future work in this area should supplement complexity metrics with features that explicitly encode syntactic structure.
Apart from these studies, many others [10
22
25
51
Syntax in the brain Traditionnlly, studies have associated small number of brain regions, usually in the left hemisphere, with syntactic processing. These include parts of the IFG, ATL and PTL. 22 3 53 54]. However, some works point to syntactic processing being distributed across the language system. Blank et al. 4 show that there significant differences in the activities of most regions of the system when phrases that are harder to parse are read compared to when easier phrases are read. Using comprehensive set of experiments, Fedorenko et al. [6] show that the entire fronto-temporal language network responds to syntax. Wehbe et al. 17 use POS and DEP tags to arrive at similar conclusions.
Previous work generally did not use naturalistic stimuli to study syntax. Instead, subjects are usually presented with sentences or even short phrases that have subtle syntactic variations or violations. Regions with activity well correlated with the presentation of such variations/violations are thought to process syntax [3]. Observations from such studies have limited scope since these variations often

---

cannot be representative of the wide range of variations seen in natural language. This is possibly why such studies report specific regions: it might be that the reported region is particularly sensitive to the exact conditions used. By using one type of stimulus which evokes only one aspect of syntactic processing, syntax might appear more localized than it really is. Our results support the hypothesis that it is instead processed in distributed fashion across the language system. We believe that our results have a wider applicability since we use naturalistic stimuli and we leave for future work the study of whether different syntactic computations are delegated to different regions.
Some studies have also doubted the importance of syntactic composition for the brain. Pylkkänen [55] proposes that there is no conclusive evidence to indicate that the brain puts a lot of weight on syntactic composition, and that even though studies (some with complexity metries) have associated certain regions like the left ATL with syntactic processing, numerous studies have later shown that the left ATI might instead be involved in more conceptually driven process. Gauthier and Levy (66) showed that BERT embeddings which were fine-tuned on tasks that removed dependency tree-based syntactic information were more reflective of brain activity than those which contained this information. In contrast, our work uses purely syntactic embeddings to show that we can indeed significantly predict many regions of the language system. We attribute these differences in conclusions to our naturalistic stimuli and word-by-word evolving representations of syntax. Pylkkänen [55 's conclusions are mostly based on studies that present a phrase with just two words (like "red boat"). Gauthier and Levy [56] use data averaged over entire sentences instead of modeling word-by-wer by-word comprehension. Since the syntactic structure of a sentence evolves with every word that is read, this approach is not necessarily adept capturing such information.
Furthermore, our analysis of the information contained in various features highlighted that our ConTreGE Incomp vectors are good at encoding complex phrase or clause-level syntactic information whereas POS and DEP tags are good at encoding local word-level information. Several regions of the brain's 's language system were predicted by ConTreGE Incomp, hinting that these areas encode complex syntactic information. Another potentially interesting observation is that including ConTreGE Incomp increases prediction performance in the PTL and IFG by more than when we include POS and DEP tags (Figure 4 but not for the ATL and the AG. These observations very loosely support the theory by Matchin and Hickok [54]. which stipulates that parts of the PTL are involved in hicrarchical and the AG is store of thematic relations between entities. This is because ConTreeGR Incomp encodes hierarchical syntactic information and word-level POS and DEP tags are very indicative of the presence of various entities (various types of nouns) and the thematic relations between entities (verbs associated with noun pairs). This hypothesis should be tested more formally in future work.
eokical-syntacti structure building. the ATL is a knowledge store of entities
Syntactic vs. semantic processing in the brain Finally, our results support the theory that syntax processing is distributed throughout the language network regions that also process semantics. This theory is supported by other studies [ 6). However, Friederici al. [53] among others argue that syntax and semantics are processed in specific and distinct regions by localizing the effects of semantic and syntactic violations.
Limitations of our approach Our first step towards studying the representations of syntax in the brain still has a few limitations. First, we cannot fully encode the information present in dependency trees using our graph embeddings-basee based approach. This is because the nodes of these trees are the words themselves and the DEP tags identify the edges between them. Our method cannot encode edge labels, it only encodes node labels. Second, we only had access to data from subjects who were asked to read English text. To determine the generality of our results, data from other experimental setups and languages will need to be analyzed using our features. Like other neural embeddings, our ConTreGE embeddings also trade off interpretability for expressivity. Although we try to deconstruct them, they are much harder to understand when compared to the simple complexity metrics or POS and DEP tags. We also try to limit the amount of semantics encoded in the embeddings. However, because of our usage of natural text, there are bound to remain some correlations between semantics and syntax that cannot be removed due the nature of human languages. Finally, the incremental top-down parser we use does not produce very good parses when compared to the more modern neural parser, limiting the concreteness of the conclusions about the Treon results.
10

---

Acknowledgments and Disclosure of Funding
LW and AJR were supported by startup funds at Carnegie Mellon University. The authors would like to thank Jennifer Williams, Mariya Toneva and Srinivas Ravishankar for useful feedback on the manuscript.
References
[1] Evelina Fedorenko and Sharon L Thompson-Schill Reworking the language network. Trends in cognitive sciences, 18(3):120-133 1262002-2222222 2014.
[2] Yosef Grodzinsky and Angela D Friederici. Neuroimaging of syntax and syntactic processing. Current opinion in neurobiology, 16(2):240-246, 2006.
[3] Angela D Friederici. The brain basis of language processing: from structure to function. Physiological reviews, (353:333733 2011.
[4] Idan Blank, Zuzanna Balewski, Kyle Mahowald, and Evelina Fedorenko. Syntactic processing distributed across the language system. Neuroimage, 127:307-323, 2016.
[5] E. Fedorenko, A. Nieto-Castanon, and N. Kanwisher. Lexical and syntactie representations in the brain: An fMRI investigation with multi-voxel pattern analyses. Neuropsychologia, 50(4): 499 5199-11,220 2012.
[6] Evelina Fedorenko, Idan Blank, Matthew Siegelman, and Zachary Mineroff. Lack of selectivity for syntax relative to word meanings throughout the language network. bioRxiv, page 477851, 2020.
[7] Yosef Grodzinsky. The neurology of syntax: Language use without broca's area. Behavioral and brain sciences, 23(1) 33(1):1-21) 2000.
[8] Jonathan Brennan, Yuval Nir, Uri Hasson, Rafael Malach, David Heeger, and Liina Pylkkänen. Syntactic structure building in the anterior temporal lobe during natural story listening. Brain and language, 120 2012.
[9] John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan R Brennan. Finding syntax in human encephalography with beam search. arXiv preprint / 2018.
[10] Roel M Willems, Stefan L Frank, Annabel D Nijhof, Peter Hagoort, and Antal Van den Bosch. Prediction during natural language comprehension. Cerebral Cortex, 26(6):2506-2516, 2015.
[11] Shohini Bhattasali, Murielle Fabre, Wen-Ming Luh, Hazem Al Saied, Mathieu Constant, Christophe Pallier, Jonathan R Brennan, R Nathan Spreng, and John Hale. Localising memory retrieval and syntactic composition: an fmri study of naturalistic language comprehension. Language, Cognition and Neuroscience, 34(4):491-510 2019.
[12] Marta Kutas and Kara D Federmeier. Thirty years and counting: finding meaning in the n400 component of the event-related brain potential (erp). Annual review of psychology, 62:621-647, 2011.
[13] Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. Where is the semantic system? critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):767-2799 2009.
[14] Tom M. Mitchell, Svetlana V. Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente Malave, Robert A. Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. Science, 320:1 1915, 2008
[15] Gustavo Sudre, Dean Pomerleau, Mark Palatucci, Leila Wehbe, Alona Fyshe, Riitta Salmelin, and Tom Mitchell. Tracking neural coding of perceptual and semantic features of concrete nouns. Neurolmage, 62:451-463, 2012.
11

---

[16] Alexander G Huth, Wendy A de Heer, Thomas L Griffiths, Frédéric E Theunissen, Jack L Gallant, Wendy a De Heer, Thomas Griffiths, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600))453--555 458,, 2016. doi: 1..1008//aurre1/66 N.tuuall
[17] Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading Subprocesses. PloS one, 1255), nov 2014. ISSN 1932-6203, doi: 10. 10.1371/journ none.0112575. URL http://dx http:://xxpplos...o. plg..oog/oooo......1 1371/journal pone. 0112575
[18] Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack L. Gallant. The representa- tion of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):7722-7736, 2019.
[19] Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals new aspects of meaning composition. bioRxiv, 2020.
[20] Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant, Encoding and decoding in fMRL. Neurolmage, 56:400-410, 2011.
[21] Jacob Devlin, Ming-Was Meig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
[22] Jonathan R Brennan, Edward P Stabler, Sarah E. Van Wagenen, Wen-Ming Luh, and John T Hale. Abstract linguistic structure correlates with temporal activity during naturalistic comprehension. Brain and Language, 157:81-92 2016.
[23] John M. Henderson, Wonil Choi, Matthew W. Lowder, and Fernanda Ferreira. Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading. Neu- rolmage, 132:293-300 300, 2016. ISSN 10959572. doi: 10.1016/j ne neuroimage.2016.022 http://dx dx doi. org/ 10. 1016/j neuroimage................ 2016. 02.000.050. 050
URL
[24] Stefan L Frank, Leun J Outen, Giulia Galli, and Gabriella Vigliocco. The erp response to the amount of information conveyed by words in sentences. Brain and language, 140:1-11, 2015.
[25] Marisa Boston, John Hale, Reinhold Kliegl, Umesh Patil, and Shravan Vasishth. Parsing costs as predictors of reading difficulty: An evaluation using the potsdam sentence corpus. The Mind Research Repository (beta), (1), 2008
[26] Evelina Fedorenko, Po-Jang Hsieh, Alfonso Vieto-Castañdo Susan Whitfeldd-Gabielli and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in individual subjects. Journal of neurophysiology. 104(2):1 177-1194, 2010.
[27] Mariya Toneva and Leila Wehbe. Interpreting and improving natural- language processing (in machines) with natural Processing Systems, pages
(in the brain). In Advances in Neural Information
2019.
[28] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In Advances in neural information processing systems, pages 6628-6637, 2018.
[29] Nicole K Speer. Jeremy R Reynolds, Khena M Swallow, and Jeffrey M Zacks. Reading stories activates neural representations of visual and motor experiences. Psychological science, 20(8): 989-999, 2009.
[30] Lyn Frazier. Syntactic complexity. Natural language parsing: Psychological, computational, and theoretical perspectives, pages 129-189, 1985.
[31] Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 170-182. Springer, 2018.
[32] Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), Melbourne, Australia, July 2018. Association for Computational Linguistics.
12

---

[33] Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. Deriving lexical and syntactic expectation based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 conference on empirical methods in natural language processing, pages 924-333 2009.
[34] Brian Roark. Probabilistic top-down parsing and language modeling. Computational linguistics, 27(2):249- 276, 2001.
[35] Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. Luminosoin- sight/wordfreq: v2.2, October 2018. /ttps://:/// 100 w01.552811//2/ 1443582
[36] Quoe V. Le and Tomas Mikolov. repprbbbuntderrreeeeenneenn of sentences and documents. CoRR, abs/1405.4053. 2014.
[37] Link 2. spaCy, en core web_sse sm model. URL batps://gittin www/coo/e/oo/ooo explosion/
spacy-modelsy wwwodl//erlleeee//aaaaaaaaeeeeee en wwwennoe_eeeeeee
wwtp_/oeeeeeeeeeeeeeeeeeeeeeeeee _ore__eb_smm2.2...
[38]
Link 1. BERT-Large, Cased: 24-layer. 1024-hidden. 16-heads, 340M parame-
ters.
URL https://storag: www.oraggggoogeapiise...o......e googleapis com wtt.l/tttttteeeeeeeeeeeeeeeeeeet porrt_mddlls/200 wateellss/211100___ cased
L-24 H-
[39] J.K. Rowling. Harry Potter and the Sorcerer's Stone. Harry Potter US. Pottermore Limited, 2012. ISBN 170171 URL http::ttt///ott http:///oooos..oloe.e.. com//oooks"idd
[40] Leila Wehbe, Aaditya Ramdas, Rebecca C Steorts, Cosma Rohilla Shalizi, et al. Regularized brain reading with shrinkage and smoothing. The Annals of Applied Statistics, 944):1997-2022, 2015.
[41] S. Nishimoto, A.T. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J.L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 2011.
[42] Y. Benjamini and Y. Hochberg, Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), pages 289-300, 1995.
[43] Christopher R. Genovese. A Bayesian time-course model for functional magnetic resonance imaging data. Journal of the American Statistical Association, 95:691-703, 2000.
[44] Link Group-level functional parcels. URL https://evlal mit. www.funcufccccc/ooooo
down wownload-pawe parcels
[45] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 4129-4138, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1111866//1111111111 URL https:/tp//....... /tt///www./// /ww.//www.ooo acleeb. org/
anthology/ N19 1419
[46] Wendy A de Heer, Alexander G Huth, Thomas I. Griffiths, Jack I. Gallant, and Frédéric E Theunissen. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27):6539-6557 2017.
[47] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. Disentangling syntax and semantics in the brain with deep networks. In International Conference on Machine Learning, pages 1336-1348. PMLR, 2021.
[48] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532 1543, 2014.
[49] Edward Stabler. Derivational minimalism. In International Conference on Logical Aspects of Computational Linguistics, pages 68-95. Springer, 1996.
[50] Edward P Stabler. Computational perspectives on minimalism. Oxford handbook of linguistic minimalism, pages 617-643, 2011.
13

---

[51] John M Henderson, Wonil Choi, Matthew W Lowder, and Fernanda Ferreira. Language structure in the brain: fixation-relate fmri study of syntactic surprisal reading. Neuroimage, 132: 293-300, 2016.
[52] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[53] Angela D Friederici, Shirley-An Rüschemeyer, Anja Hahne, and Christian J Fiebach. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cerebral cortex, 13(2):100-13) 2003.
[54] William Matchin and Gregory Hickok. The cortical organization of syntax. Cerebral Cortex, 30(3):441-1199. 2020.
[55] Liina Pylkkänen. Neural basis of basic composition: what we have learned from the red-boat studies and their extensions. Philosophical Transactions of the Royal Society B, 375(1791): 20190299, 2020.
[56] Jon Gauthier and Roger Levy. Linking artificial and human neural representations language. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 529-532 2019.
[57] Ashburner, CC Chen, G. Flandin, R. Henson, S. Kiebel, Kilner, V. Litvak, R. Moran, W. Penny, K. Stephan, et al. SPM8 manual. Functional Imaging Laboratory, Institute of Neurology, 2008.
[58] Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
[59] James Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant Pycortex: an interactive surface visualizer for fmri. Frontiers in neuroinformatics, 9:23, 2015.
14


Reviews:
[{"review_id": "vYrymiksRGH", "reviewer": "Reviewer_yLj7", "summary": "The manuscript entitled “Can fMRI reveal the representation of syntactic structure in the brain” proposed a word embedding scheme with the consideration for incomplete sentence (used to test how future sentence structure is represented in the brain) and partial parses (used to represent the multiple top-down paths of the brain in interpreting language syntaxes). Words embedded by the proposed schemes are then mapped to the corresponding fMRI signals (ensured by the experiment design where each word is strictly 0.5 second) through ridge regression. Voxels in the brain that are significantly correlated to certain combination of the syntax structures are studied in the results.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Using a more comprehensive and integrated scheme to model complex syntax structure is the main contribution and novelty of this work. Various intrinsic feature of the syntax structure (e.g., predictability in incomplete sentence) could be captured by the proposed method, which is a major improvement over the previous works. \n\nClarity: Method and result section of the proposed work is generally clearly written. With the help from the provided source code, the model should be partially replicable. Yet as the data and model are highly coupled in this work, it is difficult to fully test the model and evaluate it vs. other methods.\nSection for InConTreGE (pp. 4, line 147-158) is confusing to understand, even with the help from Fig. 1: unlike a full sentence represented by a subtree, an incomplete subtree can have arbitrary number of forms (and the corresponding measurements such as node count). How exactly ConTreGE Incomp was constructed from this near-infinite composition of words? In addition, brain regions that are correlated with ConTreGE Incomp seemed too extensive and symmetric as in Fig. 4, which in contrast to common understanding of the high-order language processing unit in the brain (which might be true given the complex task involved in this study).\n\nSignificance: The proposed work would lead to superior representability for language-related functional neuroimaging analysis than traditional functional localization studies. In addition to the proposed paradigm, more specific language-processing researches can also utilize this framework for better modeling of functional neuroimaging data. The proposed work shall be significant to a highly specialized community (cognitive neuroscience in language study).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author mentioned difference in the paradigm (naturalistic stimuli vs. designed sentences / phrases) would lead to different localization results in the discussion section, which is totally reasonable. However, they did not provide convincing premise to use the designated naturalistic paradigm: a more complex task for the subjects, such as the one adopted in this work, would introduce more external factors and confounding, which will possibly lead to inconsistent false positive regions being identified. The author is suggested to provide more neuroscientific evidence for the possible functional pathways during a reading task, in order to better elucidate the findings. \n\nA minor comment on the methodology: to the reviewer’s understanding, the ridge regression performed to map word features with fMRI signals is a 15*n->n model. The dimensionality of 15 might be too high for the ridge regression. There exist more powerful techniques to study the relationship between two sequences, where one of the sequences has higher dimensionality. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {"ethical_concerns": "A rough estimation based on the word counts in Harry Potter and the Sorcerer’s Stone (>5000 words in each chapter) and the speed of reading (0.5s for each word), each subject will undergo a scan of >40 mins plus any localization and calibration scans. Normal fMRI scan usually takes 20 mins or less. The long scan time is certainly a burden to the subjects, although written informed consent and IRB are provided. "}}, {"review_id": "QPa4kd50TZ", "reviewer": "Reviewer_djyf", "summary": "The paper proposes a new set of syntactic tree embeddings, and uses them capture additional variance in fMRI data beyond simpler complexity metrics. I think the paper setup is compelling overall, it is well-situated relative to prior work, and the results are intriguing. My primary concern is that the paper doesn't conclusively establish that these new embeddings are purely syntactic and disentangled from either complexity metrics or (more ambitiously) anything that BERT can capture, and this damages the downstream claims about neural correlates of syntactic representations. \n\nI hope this can be addressed and solved in rebuttal/revision, since I find the paper good otherwise and hope to see it at the conference. \n\nEDITED: I think the authors have reasonably addressed my concerns, and am raising my rating accordingly. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Are these embeddings truly disentangled syntax embeddings? \nThe paper makes strong claims that its embeddings need to be purely syntactic (both in the introduction starting l99 and the discussion starting l340). While much care is taken to ensure that semantics or other language features do not \"leak\" into the embedding explicitly via e.g. a lexicalized grammar, the paper does not provide an empirical demonstration of disentanglement / orthogonality. This is important in light of the paper's framing because the embeddings could nonetheless be \"incidentally\" correlated with other features, either by random chance or more realistically if there is unavoidable correlation between structures and semantics (e.g. longer sentences are used with some words/concepts but not others, which doesn't seem unreasonable). The paper should do more to demonstrate that these embeddings are truly \"purely\" syntactic (e.g. showing that on they are empirically orthogonal to the other embeddings), or moderate its claims. This is particularly important given that the distributed significance maps could equally well imply a distributed representation for syntax as a leak of other information into the embedding. \n\nTo make room for this additional demonstration, I think the introduction containing extended background work could be shortened. Even if the extended background is kept, I would recommend writing a short introduction that puts the paper's contribution front-and-center, as it is currently well into the third page of the paper. A longer background discussion can follow. In addition, I think the section beginning l285, which seems to be a sanity check of the new embeddings' ability to capture  structural information without addressing the disentangling issue above, could be moved to the appendix if additional room is needed. \n\n# Other comments\n\n- Surprisal may be unfamiliar to the broad NeurIPS audience, the paper may consider defining it if space permits. \n- I think the complicated acronyms like ConTreGE do not much to remind the reader about the content of the embedding, and are hard to match to the simpler acronyms like CC, CI, INC used in the figures. \n- The sliding window length for the embedding is first mentioned on line 188 but implicitly mentioned without definition on line 163. \n- Why do the semantic and syntactic feature dimensionalities need to be matched (l213)? \n- l214 has a stray x. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of limitations is good, though missing the concern above regarding semantic leakage. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KQj1PNeNXpR", "reviewer": "Reviewer_zeMS", "summary": "In this paper, the authors propose new representational spaces that capture syntactical information in natural language above and beyond complexity-based metrics that have been typically used in language neuroscience. First, they build representations based on the constituency tree of a sentence as every word is processed by the parser. Then, they use a subgraph embedding algorithm to convert the subtree into a 15-D representation. To test several hypothesis about the type of syntactic information being encoded, the authors look at 3 different types of subtrees- the largest subtree completed when the current word is processes, the incomplete subtrees that can explain the sequence seen thus far under PSG production rules and the set of complete parses produced by a probabilistic model that are weighted by their probabilities. The 3 subspaces are compared against 4 complexity/load-based metrics and additionally, a semantic feature space derived from BERT. They go on to build encoding models for the different feature spaces and compare them through variance partitioning. Overall, the paper finds that the 3 graph-based spaces explain additional variance across the cortex as compared to traditional complexity-based features. Further, this effect is not localized to any region. Instead, it seems to be distributed across a network that is largely explained by semantic features, suggesting that syntax and semantic share neural substrates and there exists no isolated region for syntactic processing.\n\nThe study uses publicly released fMRI data from 9 subjects reading a chapter from Harry Potter .", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper was well-written and easy to understand. While it would help to clarify how the 3 important ConTreGE features are computed for different constituents/nodes, I thought the idea of breaking down syntactic processing into 1) current subtree 2) anticipated subtree 3) all possible subtrees was thoughtful. Although the implications on language neuroscience are not novel (other work has shown or hinted that syntax and semantics likely have shared neural processing), to the best of my knowledge this is the first work demonstrating it in an encoding model setup with feature spaces that go beyond traditional effort-based metrics. \n\nSome clarifications & suggestions:\n- Re ConTreGE vectors:\n    - How many words in a sentence have different representations than POS Tags? For instance, all specifiers/heads would essentially have the same information as their POSTags but a complement or modifier might not right? It is unclear to me what the authors mean by \n> the largest subtree that is completed\nFor instance, why isn’t VP a valid subtree for _believed_ assuming that this verb has no complement? I understand that this is a secondary concern given that the representations are subgraph embeddings and thus, richer than the POSTags in many instances. But it is unclear to me how many times this is actually true and what are the rules that govern subtree selection. (My confusion was reinforced by the results of PD vs. CC in Fig. 3 & 4 + given that the random walk is still on nodes and not edges)\n- Re ConTreGE Incomp vectors:\n    - The example provided in Fig. 1C is used as a feature for _believed_ or _it_? I presumed _believed_ based on the hypothesis that the brain is already predicting future structure when it hears the word _believed_. It is unclear to me what this would look like for an already complete constituent, like _Harry_. Would this be any different from ConTReGE Comp? I agree that the idea of predictive coding (or so to speak) is interesting in terms of syntax, but do we really have the temporal resolution for this? Given that the word-space is downsampled and several of the features are very similar to the complete trees, I would like to know what the authors think.\n- Re Incremental ConTreGE: How does this compare to say, using the softmax layer of a POSTagger? [I also think it would be more demonstrative to add examples of sub-trees where the structure itself is ambiguous: Maybe like a garden path sentence?]\n- How do you deal with bidirectionally in BERT, given that the fMRI responses are generated causally?\n- I had a hard time interpreting Fig. 3:\n    - If the values are discrete from 0-5 , why not use a discrete & distinct colormap like viridis? It is not obvious to me what anything but dark green/white means.\n    - Since the subjects were coregistered to a common space, I think this figure might be underselling performance. For instance, its possible that one subject has lower prediction overall due to SNR etc.. To this end, wouldn’t plotting just the number of subjects underestimate the unique variance explained by the new feature space?\n    - What are the $R^2$ values? It seems that adding the ConTreGE features doesn’t improve performance substantially but this could be an incorrect reading of the figure. In addition to Fig. S5, I would appreciate a 3D model showing the unique variance explained for ever feature space that is added (and not the concatenation). Also, what is the significance threshold for the correlation values plotted in Fig. S5?\n    - Missing reference that this is essentially variance partitioning [deHeer et al., 2017]\n    - It seems strange to me that not enough voxels & subjects in the frontal cortex benefit from BERT when several studies (including the ones cited in this paper) have shown robust responses to semantic features in that part of the brain (especially given that none of the syntax models predict well there) Although, Fig. S5(k) suggests high correlation.\n- My comments on ease of interpretation also apply to Fig. 4 wherein it would be helpful to show the percentage of significant voxels under each model as an individual bar. (I understand that it might be too crowded; but maybe stacked bars?) \n    - I find it interesting that comparable gains are found between BERT and some of the syntactical feature spaces in Broca’s area (contained in IFG) and sPMv (contained in MFG).\n- How are the word features downsampled? Would be useful to add this to the paragraph starting in Line 229.\n- For the _bootstrap_ test in lines 251-255: I am confused if the TR blocks were sampled with replacement like in a bootstrapping procedure or were randomly permuted like in a permutation test.\n- What does the sliding window refer to in line 188?\n- It would help to be more consistent in citing work that uses encoding models for fMRI. For instance, several publications that use encoding models for fMRI (JR King’s group, Jelle Zuidema’s group, Uri Hasson’s group, Alex Huth’s group) have been elided and there is inconsistency in what publications from the references are used at specific points (lines 63-64).\n- Missing reference: Bhattasli et al., 2018 is closely related to _parser action count_ in Hale et al., 2018.\n\nI am fairly certain who the authors are from having seen a previous version of the paper + a talk but the authors did not violate anonymity in any way.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper contains information on the limitations of the approach specifically but not broad societal impacts of encoding models and neuroimaging techniques.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1FLzIS4dOi5", "reviewer": "Reviewer_KgNW", "summary": "* The authors conduct a brain encoding study testing whether fine-grained information about the content of incremental syntactic representations contribute to predictions of neural activity, over and above measures of syntactic complexity.\n* They design custom subgraph embedding representations to operationalize the idea of \"syntactic representation.\" They find that these subgraph embeddings capture additional variance in brain activity above what is predicted by complexity metrics, and that this improvement in prediction is distributed throughout the language system.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors confront the important question of the representation of syntactic content in brain activity during language processing. I appreciate their creative choice of graph embeddings as a more specific, but still somewhat theory-free, proxy for mental representations of syntax. The paper could use improvements both in interpretation of results and in organization/clarity. But I believe the authors are making a good contribution to an important conversation. I mark the paper as marginally below acceptance threshold, but would change my review if the authors can (minimally) adequately address the theoretical concerns in the \"Quality\" section below.\n\n## Quality\n\n### Are ConTreGE embeddings distinctly syntactic?\n\nThere seem to be two distinct claims that are important to the authors, and which they take their results to support:\n\n1. Syntactic representational content at a level deeper than complexity measures is present in recordings of brain activity. \\*ConTreGE\\* embeddings contain some of this representational content, and thus succeed at brain encoding over and above the predictions of complexity measures.\n2. \\*ConTreGE\\* embeddings are distinctly syntactic, and so there exist brain regions selective for syntax iff there are regions well predicted by \\*ConTreGE\\* embeddings but not by BERT embeddings.\n\nMy hunch is that while \\*ConTreGE\\* embeddings do contain meaningful syntactic content, they also contain (correlates of) semantic content. This is because constituency trees still retain information at the syntax--semantics boundary. For example, in the PTB syntactic representation it is possible to read off semantic features such as number, aspect (from correlated tense features), and modality (from the presence/absence of auxiliaries).\n\nUnder this hunch, \\#1 is valid but \\#2 is not. Overlap in well-predicted regions between the \\*ConTreGE\\* embeddings and BERT embeddings (after controlling for ConTreGE success) may be due to the fact that the \\*ConTreGE\\* embeddings are not distinctly syntactic.\n\nI would like the authors to address this question more directly. They do include (in lines 285--308) a test of the syntactic representational content of the derived ConTreGE embeddings, but this is of course not the same as demonstrating that the embeddings are distinctly syntactic. They might include more tests of representational content (or lack thereof) in the ConTreGE embeddings (e.g. visualizations of principal components of embedding space, decoding experiments, etc.), or post-hoc qualitative or quantitative evaluations of the differences between ConTreGE and BERT embeddings in real linguistic contexts.\n\nAs I understand it, this premise that ConTreGE embeddings are distinctly syntactic only matters for the inference about the overlap between syntactic and semantic content in neural representations. If there are other claims depending on this premise, I think they should also be reevaluated.\n\n### Comparing the representations\n\nI have several small qualms with interpretation of the differences between brain encoding with the different representations:\n\n1. The relative success of the Incomp representation is taken as evidence that the brain is modeling future syntactic information (lines 281--283). But Incomp can contain more context than Comp in both directions. This is evident in the Comp--Incomp contrast in Figure 1B/1C: the Incomp representation carries information about content appearing before the largest completed subtree as well as after. Because of this, I don't think you can use the relative success of Incomp to argue for representations of future syntactic content.\n2. Why do we believe that the Incomp representation should not represent uncertainty about the structure governing past content? (Why does Fig. 1C assume that \"believed\" is VBD? How is VBD chosen here over the alternative VBN as Fig. 1E makes apparent?)\n\n## Clarity\n\nThe paper could benefit from some reorganization. The introduction is extremely heavy at the moment (a full two pages). I would recommend the authors compress this section and use some of the saved space to clearly outline the embedding methods and run intrinsic analyses / visualizations on these embeddings before moving on to brain encoding work.\n\n## Other minor comments\n\n1. I'm not a neuroscientist and can't address the validity of the encoding method used. I was slightly surprised to see the authors chose to learn a mapping from all the representations triggered in a TR, rather than using a convolution based on a hemodynamic response function.\n2. Fig. 1e --- How is this a valid parse? If this is supposed to be a relative clause construction, the constituency structure is incorrect; the VBN should be a child of the NP. Also, an RC is implausible here with \"Harry\" -- you'd want a nonlexical NP such as \"the person.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper spurred a lot of discussion and back and forth with the authors, leading two reviewers to raise their scores. One reviewer thought the work is of limited interest and focused on a small community. I do not think this is grounds for reducing its score, since neuroscience is a main area in NeurIPS, definitely historically but also nowadays. 

A major issue was that the reviewers questioned whether the proposed embeddings are truly disentangled syntax embeddings. This point was partly addressed by the authors providing more analysis. However, the reviewers still would like to see a more careful and nuanced discussion of this issue, how to interpret the results in light of the syntax-semantics debate, and to make sure that limitations of the present work are discussed. 

The reviewers also mentioned problems with clarity and paper organizations -- specific questions were mostly answered, but paper organization should be improved in the next revision. 

Given the discussion, I believe the paper passes the bar for acceptance. My low confidence here is mostly because my insufficient familiarity with neurolinguistics. 
Decision: Accept (Poster)