Syntactic representations in the human brain: beyond effort-based metrics
Authors: Aniketh Janardhan Reddy
Leila Wehbe
Venue:  (2021)
Keywords: neuroscience
fMRI
syntactic representations
graph embeddings

Abstract: We are far from having a complete mechanistic understanding of the brain computations involved in language processing and of the role that syntax plays in those computations.  Most language studies do not computationally model syntactic structure, and most studies that do model syntactic processing use effort-based metrics. These metrics capture the effort needed to process the syntactic information given by every word (Brennan et al., 2012; Hale et al., 2018; Brennan et al.,2016). They can reveal where in the brain syntactic processing occurs, but not what features of syntax are processed by different brain regions. Here, we move beyond effort-based metrics and propose explicit features capturing the syntactic structure that is incrementally built while a sentence is being read.  Using these features and functional Magnetic Resonance Imaging (fMRI) recordings of participants reading a natural text, we study the brain representation of syntax. We find that our syntactic structure-based features are better than effort-based metrics at predicting brain activity in various parts of the language system. We show evidence of the brain representation of complex syntactic information such as phrase and clause structures. We see that regions well-predicted by syntactic features are distributed in the language system and are not distinguishable from those processing semantics. Our results call for a shift in the approach used for studying syntactic processing.

Under review as a conference paper at ICLR 2021
SYNTACTIC REPRESENTATIONS IN THE HUMAN BRAIN: BEYOND EFFORT-BASED METRICS
Anonymous authors Paper under double-blind review
ABSTRACT
We are far from having a complete mechanistic understanding of the brain compu- tations involved in language processing and of the role that syntax plays in those computations. Most language studies do not computationally model syntactic structure and most studies that do model syntactic processing use effort-based metrics. These metrics capture the effort needed to process the syntactic informa- tion given by every word (Brennan et al., 2012 Hale et al.) 2018 Brennan et al.. 2016). They reveal where in the brain syntactic processing occurs, but not what features of syntax are processed by different brain regions. Here, we move beyond effort-based metries and propose explicit features capturing the syntactic structure that is incrementally built while a sentence is being read. Using these features and functional Magnetic Resonance Imaging (fMRI) recordings of participants reading a natural text, we study the brain representation of syntax. We find that our syntactic structure-based features are better than effort-based metries at predicting brain activity in various parts of the language system. We show evidence of the brain representation of complex syntactic information such as phrase and clause structures. We see that regions well-predicted by syntactic features are distributed in the language system and are not distinguishable from those processing semantics. Our results call for shift in the approach used for studying syntactic processing.
INTRODUCTION
Neuroscientists have long been interested in how the brain processes syntax. To date, there is no consensus on which brain regions are involved in processing Classically, only a small number of regions in the left hemisphere were thought to be involved in language processing. More recently, the language system was proposed to involve a set of brain regions spanning the left and right hemisphere (Fedorenko & Thompson-Schil 2014). Similarly, some findings show that syntax is constrained to specific brain regions (Grodzinsky & Friederici [2006; Friederici] 2011), while other findings show syntax is distributed throughout the language system (Blank al. 2016 Fedorenko et al. 2012 2020).
The biological basis of syntax first explored through studies of the impact of brain lesions lan- guage comprehension or production (Grodziisky]] 2000) and later through non-invasive neuroimaging experiments that record brain activity while subjects perform language tasks, using methods such as functional Magnetic Resonance Imaging (IMRI) or electreencephalograpty (EEG). These experi- ments usually isolate syntactic processing by contrasting the activity between a difficult syntactic condition and an easier one and by identifying regions that increase in activity with syntactic effort (Friederici 201T). An example of these conditions is reading sentence with an object-relative clause (e.g. "The rat that the cul chased was tired"), which is more taxing than reading sentence with subject-relative clause (e.g. "The cat that chased the rot was tired"). the past decade, this approach was extended to study syntactic processing in naturalistic settings such as when reading listening to a story (Brennan et al., 2012: Hale et al. 2018 Willems et al. 2015). Because such complex material not organized into conditions, neuroscientists have instead devised effort- metries capturing the word-by-word evolving syntactic demands required to understand the material. Brain regions with activity correlated with those metries suggested to be involved in processing syntax.
We use the term effort-based metrics to refer to uni-dimensional measures capturing word-by byd-by-word syntactic demands. A standard approach for constructing syntactic effort-based metric is to assume

---

Under review as a conference paper at ICLR 2021
sentence's 's syntactic representation and estimate the number of syntactic operations performed at each word. Node Count is popular such metric. relies on constituency trees (structures that capture the hierarchical grammatical relationship between the words in sentence). While traversing the words of the sentence in order, subtrees of the constituency tree get completed; Node Count refers to the number of such subtrees that get completed at each word, effectively capturing syntactic load or effort. [Brennnn et al. (2012) use Node Count to support the theory that the Anterior Temporal Lobe (ATL) is involved in syntactic processing, Another example of an effort-based metric given by an EEG study by |aae et al. (018)) They show that parser action count (the number of possible actions a parser can take at each word) is predictive of the P600, a positive peak in the brain's electrical activity occurring around 600ms after word onset. The P600 is hypothesized to be driven by syntactic processing (to resolve incongruencies), and the results of Hale et al. (2018) align with this hypothesis.
Though effort-based metrics are a good proposal for capturing the effort involved in integrating word into the syntactie structure sentence, they are not reflective of the entire syntactie information in play. Hence, these metrics cannot be used to study the brain representation of syntactic constructs such as nouns, verbs, relationships and dependencies between words, and the complex hierarchical structure of phrases and sentences.
Constituency trees and dependency trees are the two main structures that capture a sentence's 's syntactic structure. Constituency trees are derived using phrase structure grammars that encode valid phrase and clause structure (see Figure T A) for an example). Dependency trees encode relations between pairs of words such subject-verb relationships. We representations derived from both types of trees. We derive word level dependency (DEP) labels from dependency trees, and we focus on encoding the structural information given by constituency trees since we want to analyze if the brain builds hierarchical representations of phrase structure. We characterize the syntactic structure inherent in sentence constituency trees by computing an evolving vector representation of the syntactic structure processed at each word using the subgraph embedding algorithm by Adhikari et al. (2018). We show that our syntactic structure embeddings along with other simpler syntactic structure embeddings built using conventional syntactic features such as part-of-speed better than effort-based metrics at predicting the iMRI data of subjects reading text. This indicates that representations of syntax, and not just syntactic effort, can be observed in IMRI.
(POS) tags and DEP tags are
We also address the important question of whether regions that are predicted by syntactic features are selective for syntax, meaning they only responsive syntax and not to other language properties such as semantics. To answer this question, we model the semantic properties contextual word embedding space (Devlin et al., 2018). We find that regions that predicted by syntactic features are also predicted by semantic features and thus are not selective for syntax.
words using a
Scientific questions We ask three main questions:
How can scientists construct syntactic structure embeddings that capture the syntactic structure inherent in phrases and sentences?
Are these embeddings better at predicting brain activity compared to effort-based metrics when used as inputs to encoding models?
Which brain regions are involved in processing complex syntactic structure and are they different from regions involved in semantic processing?
Contributions We make four main contributions:
We propose a subgraph embeddings- method to model the syntactic structure inherent in
phrases and sentences.
We show that effort-based metries can be complemented by syntactic structure embeddings which can predict brain activity to larger extent than effort-based metrics.
Using syntactic structure embeddings, we find some evidence supporting the hypothesis that the brain processes and represents complex syntactic information such as phrase and clause structure.
We find evidence supporting the existing hypothesis that syntactic processing appears to be distributed in the language network in regions that are not selective for syntax.
2

---

Under review as a conference paper at ICLR 2021
2 METHODS
We first describe the syntactic features used in this study and their generation. All of the features we use are incremental i.e. they are computed per word. We then describe our fMRI data analyses.
Effort-based metries We use four effort-based metrics in our analyses Node Count, Syntactic Surprisal, word frequency and word length. Node Count is an effort-based metric popular in neuroscience. To compute it, obtain the constituency tree of each sentence using the self-attentive encoder-based constituency parser by KKtaey & Klein (2018). We compute Node Count for each word as the number of subtrees that are completed by incorporating this word into its sentence. Syntactic Surprisal is another effort-based metric proposed by Roark et al. (009) and is computed using an incremental top down parser (Roark) 2001). Both of these metrics aim measure the amount of effort that is required to integrate a word into the syntactic structure of its sentence. The word frequency metric computed using the wordfreq package (Speer et al., 2018 as the Zipf frequency of a word. This is the base- 10 logarithm of the number of occurrences per billion of given word in a large text corpus. Finally, word length the number of characters in the presented word. The last two metrics approximate the amount of effort that is required to read word.
Constituency tree-based Graph Embeddings (ConTreGE) Constituency trees are a rich source of syntactic information. We build three representations of these trees that encode this information:
(a) The largest subtree which is completed upon incorporating a word into a sentence (see figure [B) is representative of the implicit syntactic information given by the word. Given that Node Count reduces all of the information present in these subtrees to just one number, it easy to see that cannot effectively capture this information. POS tags (categorize words into nouns, verbs, adjectives, etc.) also capture some of the information present in these trees as they encode phrase structure to a certain extent. But, they are incapable of completely encoding their hierarchical structure and the parsing decisions which are made while generating them. In order to better encode their structure, we first build subgraph embeddings of these completed subtrees called ConTreGE Comp vectors.
(b) We hypothesize that the brain not only processes structure seen thus far but also predicts future structure from structure it already knows. To test this, construct embeddings, simply called ConTreGE vectors, using incomplete subtrees that are constructed by retaining all the phrase structure grammar productions that are required derive the words seen till now, thereby allowing us to capture future sentence structure (in the form of future constituents) before the full sentence is read (see figure (C)). These subtrees contain leaves that are non-terminal symbols unlike complete subtrees that only have terminal symbols (words and punctuation) as leaves. In this context, non-terminal symbol a symbol that can be derived further using some rule in the phrase structure grammar (ex. NP, VP, etc.). If incomplete subtrees are more representative of the brain's 's processes, it would mean that the brain expects certain phrase structures even before the entire phrase or sentence is read, ConTreGE Comp and ConTreGE vectors need to be built using accurate constituency trees constructed using the whole sentence. Thus, we reuse the trees generated to compute Node Count to build them.
(c) Further, the brain could be computing several possible top down partial parses that can derive the words seen thus far (see figures [ (D) and (E)) and modifying the list of possible parses as future words are read. To test this hypothesis, that are representative of the most probable parses so far. For a given word, its InConTreGR. vector is computed as: v = Σ W, where W; is the subgraph embedding of a partial parse tree built by an incremental top-down parser (Roark 2001 CoLing) after reading the word and s, is the score assigned to this partial parse that is inversely proportional to the parser's 's confidence in this tree.
designed Incremental ConTreGE (InConTreGE) vectors
To effectively capture the structure of all subtrees, we encode them using the subgraph embeddings proposed by Adhikari et al.) (2018) which preserve the neighbourhood properties of subgraphs. long fixed length random walk on subgraph is generated to compute its embedding. Since consecutive nodes in random walk are neighbours, long walk can effectively inform us about the neighbourhoods of nodes in the subgraph. Each node in a walk is identified using its unique a "paragraph" where the words are the node IDs.
ID. So, random walk can be interpreted Finally, the subgraph's embedding is computed as the Paragraph Vector (Le & Mikolov 2014) of this paragraph that is representative of the subgraph's 's structure. Note that all of the subtrees of given type (complete, incomplete or partial parse) are encoded together. This ensures that all Con Comp vectors, all ConTreGE vectors and all InConTreGE vectors are in our own spaces.
3

---

Under review as a conference paper at ICLR 2021
D
E
helieved
NP
AP
VP
NP
NP
VP
NNP
VBD
NP
NNP
VBD
NP
NNP
VBD
NNP
VIN
Harry
believed
Harry
believed
Harry
betieved
Harry
believed
Figure 1: Example of complete and incomplete subtrees and two possible partial parses: Part A shows sentence's 's constituency tree generated by self-attentive encoder-based constituency parser (Kitaev & Klein 2018) using all of its words. The largest completed subtree for "believed" is shown in part and the incomplete subtree generated till "believed" is shown in part C. Incomplete subtrees are generally much deeper than complete ones. In parts D and E, we can see two possible partial parses generated by an incremental top-down parser [Roark] (00T)) only using the words till "believed". We see that the POS tag assigned to "believed" is different in the two parses.
Subtree encoded
Convert symbols numbers
Generate long random walk
Compute
Puragraph Vectors of the walk
[0.032, -1 1,2221, 3.122, -1,674
2, 10001, 10001, 3, 10002, 1,6, 1,2, 3 s....
NP
VP
5.454, 084, -7.995, 68.215,6.544
NNP
NP
2,785, 3,475, -7.997779977
Harry
believed
I0001
10002
Figure 2: Steps for encoding subtrees.
Figure iillstrates the subtree encoding process. First, every unique non-terminal in the subtrees is mapped to a unique number (ex. S is mapped to I, NP is mapped to 2, etc.) and every terminal mapped to a unique number that representative of the order which they were presented (the first presented token is mapped to 10000, the second token is mapped to 10001 and so on). We did unique number (for instance, we did not map all instances of
not map each unique terminal "Harry" to one number) because a random walk through the tree could give us word co-occurrence information and thus lead the inclusion some semantic information the vectors.
Every tree node's label is then replaced by the number it was mapped to in the previous step. The edge lists of these subtrees are supplied to the subgraph embedding generation algorithm to finally obtain 15-dimensional vectors for every presented word. The length of the random walks is set to 100000 and we use an extension of the Distributed Bag of Nodes (DBON) model proposed by & by Adhikari et al. (2018). The length of the sliding window is set to 5 and the model is trained for 20 epochs. Since Con TreGE Comp, ConTreGE and InCon' TreGGE encode information about the neighbourhoods of all nodes in the constituency trees, they can capture their hierarchical structure. Thus, brain regions predicted by these vectors are likely to be involved in building and encoding hierarchical sentence structure.
Mikolov (2014) for generating Paragraph Vectors called Seb2Vec-DBel
Punctuation We create one-hot binary vectors indicating the type of punctuation that was presented along with a word (e.g. ,) For example, sentence might have ended with "Malfoy.". In this punctuation-be based feature space, the column corresponding to will be set to for this word. While punctuation seldom considered a syntactic feature, sentence boundaries are highly correlated with changes in working memory load. These changes are bound to be great source of variability in the fMRI signal (as we will observe later). Failing account for sentence boundaries and working memory might be source of confounding that has been ignored in the literature.
Part-of-speech tags and dependency tags We use two standard word-level syntactic features POS and DEP tags. The POS tag of a word is read off previously generated constituency trees. The DEP tag of a word (ex. subject, object, etc.) correspond to its assigned role in the dependency trees of the presented sentences which were generated using the spaCy English dependency parser (2). We create one-hot binary vectors indicating the POS tag and the DEP tag of each word and concatenate them to create feature space which refer to simple syntactic structure embeddings.
4

---

Under review as a conference paper at ICLR 2021
Semantic features We adapt the vectors obtained from layer 12 of pretrained (I) cased BERT- large model (Devlin et al. 2018) to identify regions that process semantics. We use layer 12 because of previous work showing that middle layers of sentence encoders are optimal for predicting brain activity (Jain & Huth 2018 (Toneaa& & Wehbe 2019). We obtain the contextual embeddings for a word by running the pretrained model only on the words seen thus far, preventing the inclusion of future semantic information. Since a presented word can be broken up into multiple subtokens, we compute its embedding as the average of the subtokens' embeddings. Using principal component analysis (PCA), we reduce their dimensionality to 15 to match the ConTreeGE vectors' dimensionality,
IMRI data We use the fMRI data 9 subjects reading chapter 9 of Harry Potter and the Sorcerer's Stone (Rowling 2012). collected and made available by |Webbe et al. (2014). Words are presented one at a time at rate of 0.5s each. All the brain plots shown here are averages over the subjects in the Montreal Neurological Institute (MNI) space. Preprocessing details are in Appendix B.
Predicting brain activity The applicability of given syntactic feature in studying syntactic processing is determined by its efficacy in predicting the brain data described above. Ridge regression is used to perform these predictions and their coefficient of determination (R² score) measures the feature's 's efficacy. For each voxel of each subject, the regularization parameter is chosen independently. We use Ridge regression because of its computational efficiency and because of the Wehbe et al.) (2015) results showing that with such fMRI data, long the regularization parameter is chosen by cross-validation for each voxel independently, different regularization techniques lead to similar results. Indeed, Ridge regression is common regularization technique used for predictive fMRI models (Mitchell et al. 2008 Nishimoto et al. 2011 Wehbe et al. 2014 Huth et al. 2016).
For every voxel, model is fit to predict the signals Y = [y1, 1/2- Val recorded in that voxel where n is the number of time points (TR, or time to repetition). The words are first grouped by the TR in which they were presented. Then, the features of words in every group are summed form a sequence of features X = [I1, ........ xn.] aligned with the brain signals. The response measured by fMRI an indirect consequence of brain activity that peaks about seconds after stimulus onset. A common solution to account for this delay is to express brain activity as a function of the features of the preceding time points (Nishimoto et al. 2011 Wehbe et al. 2014 Huth et al. 2016). Thus, we train our models to predict any y/ using æ;- Tz -2; Ti-3 and T.-4.
We test the models a cross-validation loop: the data first split into 4 contiguous and equal sized folds. Each model uses three folds of the data for training and one fold for evaluation. We remove the data from the TRs which either precede or follow the test fold from the training set of folds. This is done to avoid any unintentional data leaks since consecutive y48 are correlated with each other because of the lag and continuous nature of the fMRI signal. The brain signals and the word features which comprise the training and testing data for each model are individually Z-scored. After training we obtain the predictions for the validation fold. The predictions for all folds are concatenated (to form a prediction for the entire experiment in which each time point is predicted from model trained without the data for that time point). Note that since all 3 ConTreGe vectors are stochastic, we construct them 5 times each, and learn different model each time. The predictions of the models are averaged together into a single prediction. The R² score is computed for every voxel using the predictions and the real signals.
We run a permutation test to test if R² scores are significantly higher than chance. We permute blocks of contiguous fMRI TRs, instead of individual TRs, to account for the slowness of the underlying hemody namic response. We choose common value of 10 TRs (Deniz et al. 2019). The predictions are permuted within fold 5000 times, and the resulting It2 scores are used as an empirical distribution of chance performance, from which the p-value of the unpermuted performance is estimated. We also run a bootstrap test to test if a model has a higher R² score than another. The difference is that in each iteration, permute (using the same indices) the predictions of both models and compute the difference of their R² and use the resulting distribution to estimate the p-value of the unpermuted difference. Finally, the Kenjammi-Hochhery False Discovery Rate correction (Benjamini & Hochberg 1995) is used for all tests (appropriate because IMRI data is considered to have positive dependence (Geooeese) 2000)). The correction is performed by grouping together all the voxel-level p-values (i.e. across all subjects and feature groups) and choosing one threshold for all of our results. The correction is done in this way since we test multiple prediction models across multiple voxels and subjects. To compute Region of Interest (ROI) statistics, left- hemisphere ROI masks for the language
5

---

Under review as a conference paper at ICLR 2021
system obtained from "sentence non-word" fMRI contrast (Fedorenko et al. 2010) obtained from (3) and mirrored to obtain the heght-hemisphere ROIs.
RESULTS
Figures 3 and summarize our results (Appendix A has the raw prediction results). Many of our features have overlapping information. POS tags include punctuation, BERT vectors have been shown to encode syntactic information (Hewitt & Manning 2019) and ConTreGE vectors, built from constituency trees, encode distinct information given by feature space, we build hierarchical feature groups in increasing order of syntactic information and test for significant differences in performance between two consecutive groups. We start with the simplest feature punctuation, and then add more complex features in order: the effort- based metrics, POS and DEP tags, one of the ConTreGE vectors and the vectors derived from BERT (which can be thought of as super-set of semantics and syntax). At each step, we test if the introduction of the new feature space leads to significantly larger than chance improvement in R².
POS tags information. To detect brain regions sensitive the
(a) (PU)
(b) (NC.PU)-(PU)
(c) [SS.PU)-(PP)
(d) (WEPU))(PUU
(c) [WI., (PU-
(0) (EF, PU)-(PU)
(g) (PD, EF, PU) -EF, PU)
(b) PD, EF PU) -IDD, EF, PU)
0
(C, PD. EF, PU) (PD, EF, PU)
(IIC, PD. EF, PU) -(PD, EF, PU)
(>.BBRT.C..DEFP... -(C. PD. EF, PU)
Figure The first plot shows the number of subjects for which a given voxel significantly predicted by punctuation (p < 0.05). The others show the number of subjects for which the difference in R² scores between two feature groups is significant (p S 0.05). Here, PU = Punctuation, NC Node Count, SS = Syntactic Surprisal, WF Word Frequency. WL Word Length, EF = All ebfort-based metrics, PD = POS and DEP Tags, CC = ConTreGE Comp, C = TreTrG INC = InCon TreGE, BERT = BERT embeddings and "(.}" indicates that these features were concatenated in order to make the predictions.
indicates a hypothesis test for the difference in R² scores between the two feature groups being larger than O. The distinct information given by syntactic structure-basse based features is more predictive of brain activity than that given by effort-based metrics. The semantic vectors are also very predictive and many well- cell-predicted regions overlap with those that are predicted by syntax.
Syntactic structure embeddings are more predictive of brain activity than effort-based metrics Figures (b)-<ee (b)-(e) show that there are small number of voxels that are predicted by the effort based metrics when taken in isolation. Figures 3 (f)-(i indicate that although the information provided by the effort metrics combined is predictive of brain activity to some degree (when controlling for punctuation), there is still considerable amount of structural information that is contained in the POS and DEP tags and in ConTreGE that predict additional portions of the activity. These results are made even clearer by Figure 4 Many voxels have significant increase in the R22 scores (above what is predicted by the effort metrics) after including POS and DEP tags and ConTreGE. We also notice that Con TreGE Comp is not predictive ConTreGE, hinting that future syntactic information helps in predicting current brain activity. Additionally, InCon TreGE is not as predictive as Con TreGE, suggesting that the top down parser might be generating partial parses that are not reflective of brain representations.
ConTreGE results suggest that complex syntactic information is encoded in the brain In this section we analyze the information in Con TreGE to interpret its brain prediction performance. We estimate how much of the constituency tree is captured by each feature by using it to predict the level

---

Under review as a conference paper at ICLR 2021
(NC,FU) -
(DD. EF. PJ) (EF. PU)
(SS. Pu)- (PU)
{CC, FD, EF, PU) -PP.. EF, PU)
(WF, PU) (PU)
{C, EF, PU) [PD,
[WL, PU) [PU)
(INC, PD, FF, PU) (PO, BF, PUJ
(Er, PU) (PU)
(DERT, C, PD, EP, PU) PD, ET, PU)
14
12
% of ficanccnncco voxels
100
tempora (Posttemp)
Temporal (AntTemp)
Angular Gyrus (Angli)
Infenice frental Gyeus (IFC;
Middle Frantal Gyrue (MFG))
Intenor Fromtl Gynus Orbita (PG0Or)
PostTemp
AntTomp
AngG
17G
MFG
IFGorb
Figure 4: Region of Interest (ROI) analysis of the prediction performance. [Left] Language system
ROIs by Fedorenkoo et al. (2010) from (3). [Right] Percentage of significantly predicted ROI voxels.
Each bar represents the average percentage across subjects and the error bars show the standard error
across subjects. We use the same abbreviations as in Figure
and see the same trends across ROIs.
Feature
Level 2
Level
Level
Level
Level
Level
Level
Level 9
Most Popular Label %
51
38,76
54,42
64.05
73.44
78.25
82.38
85.82
Node Count
51
42.6*
55.45*
64.01
73.4
78.23
82.4
85.82
Synlactic Surprisal
50.21
41.48*
54.42
64.05
73.44
78.25
82.38
85.82
Word Frequency
51
41.56*
57.13
66.58
76
80.68
84.78
88.16
Word Length
50.99
39.18
54.42
64.22
73.44
78.28
83,31
88.68*
POS and DEP tags
92.23*
71.27*
67.06*
70
77.51
82.09
86.26*
89.59*
ConTreGE Comp
66.76*
51.01*
59.52*
67.95*
77.28*
82.08*
86.26*
89.68*
ConTreCE
52.91
45,42*
57.37
67.29
76,87
82.3*
86.51
90.39
InConTreGE
52.46
4
57.61*
66.75
76.13
81.08
85.27*
89.1*
BERT Embeddings 52.18
45.73*
58.62
66.79
75.68
80.31
84.72
88.72
Table 1: 10-fold cross validation accuracies in predicting the ancestors of given word. denotes accuracies significantly above chance (tested using Wilcoxon signed-rank test, p & 0.01). POS and DEP tags best predict lower level ancestors while ConTreGE vectors best predict higher level ones.
ancestor word (in its constituency tree). We vary from to 9 and train logistic regression model for each N. Since POS tags are the level 1 ancestors of words, we start the analysis at N=2. Because there are many phrase labels, we group them into 7 larger buckets noun phrases, verb phrases, adverb phrases, adjective phrases, prepositional phrases, clauses and other miscellaneous labels. Also, a word's depth in its tree is less than N, the root is considered its level N ancestor.
Table I shows the results of this analysis. We the constituency trees generated by the Kitaev & Klein (018) parser. Given the skewed label distribution, the optimal strategy for a predictor that takes random noise input always output the majority class ancestor that level. Chance performance thus equal to the frequency of the majority label. The effort-based metrics are not as predictive as ConTreGE any level. POS and DEP tags are predictive of labels at all levels and produce the highest accuracies for lower levels. The InConTreGG vectors are not as predictive as ConTreGE or ConTreGE Comp, hinting that the top down parser might not be very accurate. ConTreGE is the best predictor of higher level ancestors but CrnTreGE Comp is better than Con at predicting lower level ancestors. This may be because graph embeddings of tree tend to capture more of the information near the tree's root (a random walk through somewhat balanced tree is likely to contain more occurrences of nodes near the root). ConTreGE Comp vectors, created from shallow complete trees, likely over-represent lower level ancestors while CrnTreGE vectors, created from relatively deeper trees, likely over-represent higher level ancestors. Given that Con TreGE is predictive of brain activity and contains information about the higher level ancestors of word, this suggests that the brain represents complex hierarchical syntactic information such as phrase and clause structure.
Syntax and semantics processed in distributed way in overlapping regions across the language system Our results indicate that syntactic and semantic information are processed in distributed fashion across the language network. Most of the regions in the language system
7

---

Under review as a conference paper at ICLR 2021
better predicted by the BERT embeddings after controlling for all our other feature spaces, and these regions overlap with the regions that are predicted by the syntactic feature spaces. While the BERT embeddings include both semantic and syntactic information, it is likely that the semantic information is at least partially predictive of the brain activity, given that we have already controlled for a lot of syntactic information.
4 DISCUSSION AND RELATED WORK
Syntactic representations Apart from Brennan et al. (2012) 2) and Hale et al. (2018). many others
(Brennan et al.] 2016 Henderson et al., 2016 Frank et al. 2013 Boston et al., 2008 Willems et al. 2015) use effort-based metrics study syntactic processing during natural reading listening. However, a few studies do explicitly encode syntactic structure: Wehhe et al. (2014) find that POS and features. Moving away from popular approaches that are dependent on effort-basee metrics, we extended the work of approach to explicitly capture the syntactic information provided by constituency trees. Our results showed that these explicit features have substantially more information that predictive of brain activity than effort based metrics. Given these results, we believe that future work in this area should supplement effort-based metrics with features that explicitly encode syntactic structure.
DEP tags are the most predictive out of a set of word, sentence and discourse-
Wehbe et al. (2014) by developing a novel graph embeddings-bass
Syntax in the brain Traditionally, studies have associated a small number of brain regions, usually in the left hemisphere, with syntactic processing. These include parts of the inferior frontal gyrus (IFG), ATL and Posterior Temporal Lobe (PTL) (Grodzinsky & Friederici 2006 Friederici 1
Friederici et al. 2003 Matchin & Hickok 2020). However, some works point to syntactic processing being distributed across the language system. Blank et al. (2016) shows that significant differences in the activities of most of the language system are greater when reading hard to parse sentences than POS and DEP tags to arrive at similar conclusions.
easier phrases Wehbe et al. (2014)
Previous work generally did not use naturalistic stimuli to study syntax. Instead, subjects are usually presented with sentences even short phrases that have subtle syntactic variations or violations. Regions with activity well correlated with the presentation of such variations/violation are thought process syntax (Friederiei, 201T). Observations from such studies have limited scope since these variations often cannot be representative of the wide range of variations seen in natural language. This possibly why such studies report specific regions: might be that the reported region is particularly sensitive to the exact conditions used. By using one type of stimulus which evokes only one aspect of syntactic processing, syntax might appear more localized than it really is. Our results support the hypothesis that it is instead processed in distributed fashion across the language system. We believe that our results have a wider applicability since we use naturalistic stimuli and we leave for future work the study of whether different syntactic computations are delegated to different regions.
Some studies have also doubted the importance of syntactic composition for the brain. Pylkkänen (2020) proposes that there is no conclusive evidence to indicate that the brain puts lot of weight on syntactic composition, and that even though studies (some with effort- metrics) have associated certain regions like the left ATL with syntactic processing, numerous studies have later shown that the left ATL might instead be involved in a more conceptually driven process. Gauthier & Levy (2019) showed that BERT embeddings which were fine-tuned on tasks that removed dependency tree-based syntactic information were more reflective of brain activity than those which contained this information. In contrast, our work uses purely syntactic embeddings to show that we can indeed significantly predict many regions of the language system. We attribute these differences in conclusions to our naturalistic stimuli and word-by-word evolving representations of syntax. Pylkkänen (2020)'s conclusions are mostly based on studies that present a phrase with just two words (like "red boat"). Gauthier Levy] (2099) use data averaged over entire sentences instead of modeling word-by-word comprehension. Since the syntactic structure of a sentence evolves with every word that is read, this approach is not necessarily adept at capturing such information.
Furthermore, our analysis of the syntactic information contained in various features highlighted that our ConTre vectors are good at encoding complex phrase or clause-level syntactic information whereas POS and DEP tags are good at encoding local word-level syntactic information. Several regions of the brain's 's language system were predicted by Con TreGG, hinting that the brain does indeed encode complex syntactic information. Another potentially interesting observation is that
8

---

Under review as a conference paper at ICLR 2021
including ConTreGE increases prediction performance in the PTL and IFG by more than when we include POS and DEP tags (Figure[4) but not for the ATL and the Angular Gyrus (AG). These observations very loosely support the theory by Matchin & Hickok (2020) that parts of the PTL are involved in hierarchical lexical-syntactic structure building, the ATT. is a knowledge store of entities and the AG is store of thematic relations between entities. This is because ConTreGE encodes hierarchical syntactic information and word-level POS and DEP tags are very indicative of the presence of various entities (various types of nouns) and the thematic relations between entities (verbs associated with noun pairs). This hypothesis should be tested more formally in future work.
We also observe that ConTreGE is more predictive than ConTreGE Comp and InConTreGE with the latter two being very weakly predictive. Thus, future syntactic information appears to be very useful while predicting BOLD signals, indicating that that the brain anticipates the eventual sentence structure while reading to a more accurate extent than an incremental top down parser.
Syntactic vs. semantic processing in the brain Finally, our results support the theory that syntax processing is distributed throughout the language network in regions that also process semantics. This theory supported by other studies (Fedorenko et al. 2012; Blank et al. 2016 Fedorenko et al. 2020). However, FFiederici et al. (2003) among others argue that syntax and semantics are processed in specific and distinct regions by localizing the effects semantic and syntactic violations. Again, these differences might be due to the specialized stimuli and high statistical thresholds that only reject the null hypotheses in the regions with the strongest effect size, thereby precisely identifying small regions. A less conservative threshold might have revealed a more distributed pattern without leading to type I errors.
REFERENCES
Link URL BERT-Large, Cased: 24-layer, 1024-hidden, 16-heads, 340M parameters.
L-24 H-1024 A-16.
Link 2.
spaCy, en core webe_weeb_aeeeee sm model. URL https:/ wtt.cccc///pppeeee http://tttuu.ccom/epplooooooo
s spacy peacy-modes mode 1s hew.ss/reeelssslleeeeeeeeeeessee waasee//taaaaae aacr//n__coorrr core wttee_orreeeeeeeeeeeeeeee web_sm-2.2.7
Link 3. Group-level functional parcels. download-parcell
eddubuuu func oc/
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 170-182.
Springer, 2018.
J. Ashburner, CC Chen, G. Flandin, R. Henson, S. Kiebel, J. Kilner, V. Litvak, R. Moran, W. Penny, K. Stephan, et al. SPM8 manual. Functional Imaging Laboratory, Institute of Neurology, 2008
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), pp. 289-300, 1995.
Idan Blank, Zuzanna Balewski, Kyle Mahowald, and Evelina Fedorenko. Syntactic processing is distributed across the language system. Neuroimage, 127:307-323. 2016.
Marisa Boston, John Hale, Reinhold Kliegl, Umesh Patil, and Shravan Vasishth. Parsing costs as predictors of reading difficulty: An evaluation using the potsdam sentence corpus. The Mind Research Repository (beta), (1), 2008
Jonathan Brennan, Yuval Nir, Uri Hasson, Rafael Malach, David J Heeger, and Liina Pylkkänen. Syntactic structure building in the anterior temporal lobe during natural story listening. Brain and lunguage, 163 2012.
Jonathan R Brennan, Edward P Stabler, Sarah E Van Wagenen, Wen-Ming Luh, and John T Hale. Abstract linguistic structure correlates with temporal activity during naturalistic comprehension. Brain and Language, 157:81-94 2016.

---

Under review as a conference paper at ICLR 2021
Fatma Deniz, Anwar O Numez-Elizalde Alexander G Huth, and Jack L Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):772--7736 2019.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18 2018.
E. Fedorenko, A. Nieto-Castanon, and N. Kanwisher. Lexical and syntactic representations in the brain: An fMRI investigation with multi-voxel pattern analyses. Neuropsyccoolgia, 50(4):499-513, 2012.
Evelina Fedorenko and Sharon I. Thompson-Schill. Reworking the language network. Trends in cognitive sciences, 18(3):120-126, 2014.
Evelina Fedorenko, Po-Jang Hsich. Alfonso Nieto- Castañón, Susan Whitield-Gabrrili and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in individual subjects. Journal of neurophysiology. 104(2)1117-11194 2010.
Evelina Fedorenko, Idan Blank, Matthew Siegelman, and Zachary Mineroff. Lack of selectivity for syntax relative word meanings throughout the language network. bioRxiv, pp. 477851, 2020.
Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
Stefan Frank, Leun J Otten, Giulia Galli, and Gabriella Vigliocco. The erp response to the amount of information conveyed by words in sentences. Brain and language, 140:1-11, 2015.
Angela Friederici. The brain basis of language processing: from structure to function. Physiological reviews, 91(4):1357-1392, 2011.
Angela D Friederici, Shirlcy-Am Rüschemeyer, Anja Hahne, and Christian J Fiebach. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cerebral cortex, 13(2):170-177 2003.
James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive surface visualizer for fmri. Frontiers in neuroinformatics, 9:23, 2015.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 529-539, 2019.
Christopher Genovese. Bayesian time-course model for functional magnetic resonance imaging data. Journal of the American Statistical Association, 95:691-703, 2000.
Yosef Grodzinsky. The neurology of syntax: Language use without broca's 's area. Behavioral and brain sciences, 23(1):1-21, 2000.
Yosef Grodzinsky and Angela Friederici. Neuroimaging of syntax and syntactic processing. Current opinion in neurobiology, 16(2):240-24 246, 2006.
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan R Brennan. Finding syntax in human encephalography with beam search. arXiv preprint arXiv:1806.04127 2018.
John M Henderson, Wonil Choi, Matthew W Lowder, and Fernanda Ferreira. Language structure in the brain: A fixation-related fmri study of syntactic surprisal in reading. Neuroimage, 132: 293-300, 2016.
John Hewitt and Christopher D. Manning, A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4129 4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 18653vv/ 16533/1//N191144 1419, htl.htps:: www. a web. org. www.bb.rrgg/tthhooog....ooo 9449
10

---

Under review as a conference paper at ICLR 2021
Alexander G Huth, Wendy A de Heer, Thomas L Griffiths, Frédéric E Theunissen, Jack L Gallant, Wendy a De Heer, Thomas Griffiths, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-455 2016. doi: 10.1038/maturel77377 Natural.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In Advances in neural information processing systems, pp.
2018.
Nikita Kitaev and Dan Klein. Constituency parsing with self aelf-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), Melbourne, Australia, July 2018. Association for Computational Linguistics.
Quoc V. Le and Tomas Mikolov. Distributedd represttto sentences and documents. CoRR, Abs/1405.4055 2014. URL htt.attp//aaxtvv.orr/ab///////// 4053
William Matchin and Gregory Hickok. The cortical organization of syntax. Cerebral Cortex, 30(3):
1481-1498, 2020.
T.M. Mitchell, S.V. Shinkareva, A. Carlson, K.M. Chang, V.I., Malave, R.A. Mason, and M.A. Just. Predicting human brain activity associated with the meanings nouns. Science, 320(5880):
1191-1195, 2008.
S. Nishimoto, A.T. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J.L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 2011.
Liina Pylkkänen. Neural basis of basic composition: what we have learned from the red-boat studies and their extensions. Philosophical Transactions of the Royal Society B, 375(1791):2019022)) 2020.
Brian Roark. Probabilistic top-down parsing and language modeling, Computational linguistics, 27 (2):249-276, 2001.
Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. Deriving lexical and syntactic expectation-based based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 conference on empirical methods in natural language processing, pp. 324-333, 2009.
J.K. Rowling. Harry Potter and the Sorcerer's Stone. Harry Potter US. Pottermore Limited, 2012. ISBN 9781781 100271. URL/http://books google com/ wwtp.//oookkkkl..ooooooaaaaaa
Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. Luminosoinsggh//wordlggy v2.2, October 2018 URL htttps://doi...... /rt.///o..oo/oo////
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural processgn (in the brain). In Advances in Neural Information Processing Systems, pp. 14928-14938, 2019.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading Subpro- cesses. PloS one, 9(1 : 12575, nov 2014. ISSN 1932-6203. doi: 10.137 1 Sattrnaalppooe000011 pone.00 125755 URL htllhttt///dd plos org/ / 10. / pone 0112575
Leila Wehbe, Aaditya Ramdas, Rebecca Steorts, Cosma Rohilla Shalizi, et al. Regularized brain reading with shrinkage and smoothing. The Annals of Applied Statistics, 999):1997-209 2015.
Roel M Willems, Stefan L Frank, Annabel D Nijhof, Peter Hagoort, and Antal Van den Bosch. Prediction during natural language comprehension. Cerebral Cortex, 2666):20062257 2015.
11


Reviews:
[{"review_id": "ih_E0CTCQyK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper derives various types of graph embeddings to encode aspects of syntactic information that the brain may be processing during real-time sentence comprehension. These embeddings, along with indicators of punctuation, POS and dependency tags, and BERT embeddings, are used to predict brain activity recorded via fMRI. The authors argue that this is an improvement over use of effort-based metrics to predict brain activity, as these embeddings contain richer information than is captured by distilling down to a single measure of effort. They show that various brain regions are significantly better predicted by the syntactic embeddings than by the effort-based metrics and POS+dependency indicators. BERT embeddings, however, prove to be a better predictor (than syntactic and other predictors) across much more substantial areas of activity. \n\nI'm all for leveraging representations from NLP models to ask questions about the brain, and some of the patterns identified here are interesting, but I don't think this paper has arrived at sufficiently impactful takeaways to merit publication as yet. The main concrete conclusion drawn from these analyses is that complex syntactic information is encoded in the brain. But this really isn't a particularly disputed claim, so it's definitely underwhelming as a takeaway. The sensitivity of the brain to hierarchical syntax is also an underlying assumption of syntactically-grounded effort-based metrics, so although these graph embeddings capture richer information, if they only give us the conclusion that the brain is doing syntax, then they have not really given us new information. A related conclusion made in the paper is that syntactic information is represented in a distributed fashion, but the effort-based metrics seem to suggest a similar conclusion (if I'm correctly interpreting Fig 3f), so again this is not unique to the proposed representations. I do recognize that the proposed syntactic representations are predictive of activity over and above the effort-based metrics in some voxels, but it's not clear exactly what we learn from this fact.\n\nA secondary conclusion made in the paper is that regions that process syntax are not specialized for syntax. This conclusion seems to be made based on the fact that BERT embeddings are stronger predictors than the syntactic embeddings in many of the regions in which the syntactic embeddings outperformed other predictors. However, as the authors acknowledge, BERT embeddings also encode syntactic information, and this makes it more difficult to interpret this pattern of results. It seems that the stronger performance of BERT embeddings could just as easily be attributable to better/richer encoding of relevant syntactic information as to encoding of semantic information.  What the results seem to show is simply that the BERT embeddings are better predictors of brain activity than any of the other representations used, so the question then raised is what exactly the BERT embeddings capture that the brain activity is also sensitive to. \n\nI'll also say that a downside of the graph embeddings is that they seemingly reduce transparency relative to the effort-based metrics -- I'm certainly willing to believe that they encode richer information, but it doesn't seem clear precisely what information they are adding.\n\nAll in all, I think this is an interesting line of work, but I'm not convinced that we come away having learned something impactful, both because some of the proposed conclusions answer questions that aren't really at-issue / aren't really uniquely addressed by the proposed representations, and also because some conclusions are drawn that don't seem to follow clearly from the observed results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_PNiHjH12Ne", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a neuroimaging study investigating the way syntax is represented. The authors compare models that encode syntax with fMRI data. They find that syntax and semantics are computed/represented in overlapping brain regions and that \"complex\" syntactic information is decodable.\n\nI liked this paper and think it is valuable in, amongst other things, furthering the theoretical position that the dichotomy between semantics and syntax is a more conceptual/high-level one than can be found at the level of neuroimaging data. \n\nThe authors give an exposition of their research questions, however I think these can be phrased even more clearly in some cases. For example, for Q1 do they mean for humans in general (as in in the brain) or do they mean us as scientists researching human cognition? For Q2, do they mean they will use a model-based fMRI analysis? For Q3, is this multivariate or univariate? Just adding those short phrases or words to the research questions will help situate the reader, in my opinion.\n\nWhy is it in and of itself surprising that (complex) syntax is encoded in the brain? In other words: \"Several regions of the brain’s language system were predicted by ConTreGE, hinting that the brain does indeed encode complex syntactic information.\" — why would \"the brain\" not? Please do not get me wrong, this is obviously important/required to be shown but the research herein actually has even more value (or could have) and can be framed and discussed as such. Surely, the interesting results (since we know from other sources and common sense that syntax is, has to be, encoded in the brain somewhere since it plays a role in cognition) is the actual relationships of the results to the overarching theory and should be foregrounded more. \n\nA potentially useful theory paper is, and which might interest the authors: https://doi.org/10.1162/jocn_a_01552\n\nMinor point: the in-text citations would look better without double brackets — which is easy to fix in LaTeX.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACew4UjMjqs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to propose a parse tree embedding that correlates with brain activations better than existing measures on sentences. It is an extremely important topic as it can draw the link between Artificial NN and real NN on the problem of syntactic processing.\nHowever, the paper leaves with a major quesiton: why? \nWhy the proposed parse tree embedding model is a good model. There is a wide range of models embedding parse trees, e.g. RecursiveNN, TreeLSTM and Distributed Tree Kernel. All these models are embedding trees in different ways. Why the proposed model should be closer to the brain activity? This should be definetly clarified in the description of the parse tree embedder. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "1obJkmhgkOV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe manuscript focuses on understanding the features of syntax that are processed by different brain regions as captured by fMRI. The proposition is to move beyond effort-based metrics to subgraph embeddings for modeling syntactic structure. In addition, the approach focuses on natural reading and incremental models while a sentence is being read. I liked reading the paper and it reports interesting results. The advantage of the research is that it uses natural reading stimuli, but that also comes with disadvantages for supporting the conclusions of the manuscript (see details below).\n\nGeneral comments\n\nThe research questions and contributions are clearly written. I think the authors could milden some of the statements that are not considered to be really novel findings of the present manuscript. \n\nIt is known that syntactic processing and semantic processing are mixed and that several brain regions contribute to constitute an understanding of language (see e.g. 1,2). It is also known that the brain not only processes structure seen thus far but also predicts future structure. However, this, and other measures, such as syntactic violations, information gain, etc. has been found already in the early studies (see 3,4,5) and more recently in natural reading (see 6). Maybe the authors could be a bit more specific in what exactly is novel in the present manuscript.  \n\nMethods\n\nI did not fully understand the data split for training and testing. It states:  “is first split into 4 contiguous and equal\nsized folds. Each model uses three folds of the data for training and one fold for evaluation.” Now, if the data is from natural reading of text, does this mean that you are using samples that occur both before and after a particular word is being read? If so, does this affect the results as evidence from other (latter) parts of the sentences can be used to draw predictions for words earlier in the sentence? Prior to this, it is stated that you only use x-1,…x-4 for prediction and that “(to form a prediction for the entire experiment in which each time point\nis predicted from a model trained without the data for that time point)”. But is there now a chance that something in x+n would be in the training data for the models? Wouldn’t this mean that some data would “leak” in x+n and would be in the training set for a particular sample? Maybe I didn’t correctly understand the setting, but I would like to see a better explanation of how exactly the split is done and test and training sets are constructed, and why this is the correct way for the particular analysis conducted. \n\n\nThere are many models tested and I hope the authors are correcting for multiple testing; it is stated that Benjamni-Hochberg False Discovery Rate correction is applied, but it is not entirely clear how this is done. \n\nI did not understand why principal component analysis (PCA) was used to reduce the dimensionality to 15. Why would the dimensionality need to be the same? How does this affect the results?\n\nConclusions\n\nThe conclusions of the paper rely on the advantages of naturalistic sentence reading, but the claims that more controlled experiments that have revealed specific regions responsible for syntactic processing would be less important, are not supported by the results presented in the manuscript. There was nothing in the study setting that would have controlled for other effects. For example, what if there were some other factors in the sentences that explain the effects. What if the other model predicts some other features that are correlated with syntax structure, such as more complex words, words that carry more information, more frequent, rare, short, or long words? I think these cannot be fully excluded and I would have liked to see some other results than only the predictions of the models to confirm that the prediction study can be considered valid. Without a proper control condition or other analysis of the data, it is not evident that all of the conclusions hold.\n\n[1] https://www.sciencedirect.com/science/article/pii/S1053811915011064?casa_token=uVeVeMM5HCwAAAAA:Uj7IsuC-Kf6xwHQ-Rgs8RhxTl8A_PID_2fVStTqnuTA8Lshjb8Lil-iDOhyHJgADMGhHtjDM8kE\n\n[2] https://www.nature.com/articles/s41467-019-08848-0\n\n[3] https://www.sciencedirect.com/science/article/pii/S1053811916001592?casa_token=neHunii1ASwAAAAA:P1lkCeo5kDULBuy_6VXAjFN2gx2xK-algZXjXWaRgNxpnhGnWA3y_Vh_ey4TvynL9-CZOHHrkI0\n\n[4] https://www.sciencedirect.com/science/article/pii/S0093934X14001515\n\n[5] https://www.mitpressjournals.org/doi/abs/10.1162/089892903322370807?casa_token=SqKOci8rCScAAAAA:rvso0ntTEMzNym_FhyH-ylSilW0qRHjEiBLrFbXlV703WHLOTTZ4G3xHVHhrxRiIBXLB9PqadvV3\n\n[6] https://www.nature.com/articles/s41598-020-63828-5\n\n[7] https://www.sciencedirect.com/science/article/pii/S1053811916001592?casa_token=WtO6Twcn9yEAAAAA:6aeIl_agLqFBsaViu4Audy7KdHJKtELPp6TF2KT0stCpng80APTfD7V1rZu0gpzqDqFp8ZKHy2E", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the brain's activity in response to language, specifically targeting the signatures of syntax in the brain.  The authors specifically investigate the signatures of specific syntactic elements against the "typical" effort based syntax measures from some previous work.  

The title and abstract of the paper are clear and compelling, but the text of the paper muddies the message and this was expressed in the reviews.  There may be some debate in the literature as to if syntax and semantics are dissociable, and to what degree we can actually measure syntax in the brain, but I (and your reviewers) have trouble believing that any one actually thinks there is *no* syntax representations in the brain.  Certainly this is not a claim made by either the Federenko or Pylkkanen papers the authors cite. Federenko says "lexico-semantic and syntactic processing are deeply inter-connected and perhaps not separable" but doesn't claim that the brain doesn't "do" syntax. Pylkkanen says "Syntax in the brain is necessary to explain the fact that humans are exquisitely skilled at judging syntactic well-formedness, even for sentences that have no coherent meaning."

I suggest this paper either rephrase the arguments, more clearly articulate the issues they wish to address, or find another venue where the reviewers might be more read to debate *if* syntax is encoded in the brain.  That seems outside of the scope of ICLR.
Decision: Reject