Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper derives various types of graph embeddings to encode aspects of syntactic information that the brain may be processing during real-time sentence comprehension. These embeddings, along with indicators of punctuation, POS and dependency tags, and BERT embeddings, are used to predict brain activity recorded via fMRI. The authors argue that this is an improvement over use of effort-based metrics to predict brain activity, as these embeddings contain richer information than is captured by distilling down to a single measure of effort. They show that various brain regions are significantly better predicted by the syntactic embeddings than by the effort-based metrics and POS+dependency indicators. BERT embeddings, however, prove to be a better predictor (than syntactic and other predictors) across much more substantial areas of activity. 

I'm all for leveraging representations from NLP models to ask questions about the brain, and some of the patterns identified here are interesting, but I don't think this paper has arrived at sufficiently impactful takeaways to merit publication as yet. The main concrete conclusion drawn from these analyses is that complex syntactic information is encoded in the brain. But this really isn't a particularly disputed claim, so it's definitely underwhelming as a takeaway. The sensitivity of the brain to hierarchical syntax is also an underlying assumption of syntactically-grounded effort-based metrics, so although these graph embeddings capture richer information, if they only give us the conclusion that the brain is doing syntax, then they have not really given us new information. A related conclusion made in the paper is that syntactic information is represented in a distributed fashion, but the effort-based metrics seem to suggest a similar conclusion (if I'm correctly interpreting Fig 3f), so again this is not unique to the proposed representations. I do recognize that the proposed syntactic representations are predictive of activity over and above the effort-based metrics in some voxels, but it's not clear exactly what we learn from this fact.

A secondary conclusion made in the paper is that regions that process syntax are not specialized for syntax. This conclusion seems to be made based on the fact that BERT embeddings are stronger predictors than the syntactic embeddings in many of the regions in which the syntactic embeddings outperformed other predictors. However, as the authors acknowledge, BERT embeddings also encode syntactic information, and this makes it more difficult to interpret this pattern of results. It seems that the stronger performance of BERT embeddings could just as easily be attributable to better/richer encoding of relevant syntactic information as to encoding of semantic information.  What the results seem to show is simply that the BERT embeddings are better predictors of brain activity than any of the other representations used, so the question then raised is what exactly the BERT embeddings capture that the brain activity is also sensitive to. 

I'll also say that a downside of the graph embeddings is that they seemingly reduce transparency relative to the effort-based metrics -- I'm certainly willing to believe that they encode richer information, but it doesn't seem clear precisely what information they are adding.

All in all, I think this is an interesting line of work, but I'm not convinced that we come away having learned something impactful, both because some of the proposed conclusions answer questions that aren't really at-issue / aren't really uniquely addressed by the proposed representations, and also because some conclusions are drawn that don't seem to follow clearly from the observed results.
review_title: Interesting work, but conclusions yield minor impact / may not follow from results

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to propose a parse tree embedding that correlates with brain activations better than existing measures on sentences. It is an extremely important topic as it can draw the link between Artificial NN and real NN on the problem of syntactic processing.
However, the paper leaves with a major quesiton: why? 
Why the proposed parse tree embedding model is a good model. There is a wide range of models embedding parse trees, e.g. RecursiveNN, TreeLSTM and Distributed Tree Kernel. All these models are embedding trees in different ways. Why the proposed model should be closer to the brain activity? This should be definetly clarified in the description of the parse tree embedder. 
review_title: Official Blind Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a neuroimaging study investigating the way syntax is represented. The authors compare models that encode syntax with fMRI data. They find that syntax and semantics are computed/represented in overlapping brain regions and that "complex" syntactic information is decodable.

I liked this paper and think it is valuable in, amongst other things, furthering the theoretical position that the dichotomy between semantics and syntax is a more conceptual/high-level one than can be found at the level of neuroimaging data. 

The authors give an exposition of their research questions, however I think these can be phrased even more clearly in some cases. For example, for Q1 do they mean for humans in general (as in in the brain) or do they mean us as scientists researching human cognition? For Q2, do they mean they will use a model-based fMRI analysis? For Q3, is this multivariate or univariate? Just adding those short phrases or words to the research questions will help situate the reader, in my opinion.

Why is it in and of itself surprising that (complex) syntax is encoded in the brain? In other words: "Several regions of the brain’s language system were predicted by ConTreGE, hinting that the brain does indeed encode complex syntactic information." — why would "the brain" not? Please do not get me wrong, this is obviously important/required to be shown but the research herein actually has even more value (or could have) and can be framed and discussed as such. Surely, the interesting results (since we know from other sources and common sense that syntax is, has to be, encoded in the brain somewhere since it plays a role in cognition) is the actual relationships of the results to the overarching theory and should be foregrounded more. 

A potentially useful theory paper is, and which might interest the authors: https://doi.org/10.1162/jocn_a_01552

Minor point: the in-text citations would look better without double brackets — which is easy to fix in LaTeX.

review_title: Looking for syntax correlates in fMRI data

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The manuscript focuses on understanding the features of syntax that are processed by different brain regions as captured by fMRI. The proposition is to move beyond effort-based metrics to subgraph embeddings for modeling syntactic structure. In addition, the approach focuses on natural reading and incremental models while a sentence is being read. I liked reading the paper and it reports interesting results. The advantage of the research is that it uses natural reading stimuli, but that also comes with disadvantages for supporting the conclusions of the manuscript (see details below).

General comments

The research questions and contributions are clearly written. I think the authors could milden some of the statements that are not considered to be really novel findings of the present manuscript. 

It is known that syntactic processing and semantic processing are mixed and that several brain regions contribute to constitute an understanding of language (see e.g. 1,2). It is also known that the brain not only processes structure seen thus far but also predicts future structure. However, this, and other measures, such as syntactic violations, information gain, etc. has been found already in the early studies (see 3,4,5) and more recently in natural reading (see 6). Maybe the authors could be a bit more specific in what exactly is novel in the present manuscript.  

Methods

I did not fully understand the data split for training and testing. It states:  “is first split into 4 contiguous and equal
sized folds. Each model uses three folds of the data for training and one fold for evaluation.” Now, if the data is from natural reading of text, does this mean that you are using samples that occur both before and after a particular word is being read? If so, does this affect the results as evidence from other (latter) parts of the sentences can be used to draw predictions for words earlier in the sentence? Prior to this, it is stated that you only use x-1,…x-4 for prediction and that “(to form a prediction for the entire experiment in which each time point
is predicted from a model trained without the data for that time point)”. But is there now a chance that something in x+n would be in the training data for the models? Wouldn’t this mean that some data would “leak” in x+n and would be in the training set for a particular sample? Maybe I didn’t correctly understand the setting, but I would like to see a better explanation of how exactly the split is done and test and training sets are constructed, and why this is the correct way for the particular analysis conducted. 


There are many models tested and I hope the authors are correcting for multiple testing; it is stated that Benjamni-Hochberg False Discovery Rate correction is applied, but it is not entirely clear how this is done. 

I did not understand why principal component analysis (PCA) was used to reduce the dimensionality to 15. Why would the dimensionality need to be the same? How does this affect the results?

Conclusions

The conclusions of the paper rely on the advantages of naturalistic sentence reading, but the claims that more controlled experiments that have revealed specific regions responsible for syntactic processing would be less important, are not supported by the results presented in the manuscript. There was nothing in the study setting that would have controlled for other effects. For example, what if there were some other factors in the sentences that explain the effects. What if the other model predicts some other features that are correlated with syntax structure, such as more complex words, words that carry more information, more frequent, rare, short, or long words? I think these cannot be fully excluded and I would have liked to see some other results than only the predictions of the models to confirm that the prediction study can be considered valid. Without a proper control condition or other analysis of the data, it is not evident that all of the conclusions hold.

[1] https://www.sciencedirect.com/science/article/pii/S1053811915011064?casa_token=uVeVeMM5HCwAAAAA:Uj7IsuC-Kf6xwHQ-Rgs8RhxTl8A_PID_2fVStTqnuTA8Lshjb8Lil-iDOhyHJgADMGhHtjDM8kE

[2] https://www.nature.com/articles/s41467-019-08848-0

[3] https://www.sciencedirect.com/science/article/pii/S1053811916001592?casa_token=neHunii1ASwAAAAA:P1lkCeo5kDULBuy_6VXAjFN2gx2xK-algZXjXWaRgNxpnhGnWA3y_Vh_ey4TvynL9-CZOHHrkI0

[4] https://www.sciencedirect.com/science/article/pii/S0093934X14001515

[5] https://www.mitpressjournals.org/doi/abs/10.1162/089892903322370807?casa_token=SqKOci8rCScAAAAA:rvso0ntTEMzNym_FhyH-ylSilW0qRHjEiBLrFbXlV703WHLOTTZ4G3xHVHhrxRiIBXLB9PqadvV3

[6] https://www.nature.com/articles/s41598-020-63828-5

[7] https://www.sciencedirect.com/science/article/pii/S1053811916001592?casa_token=WtO6Twcn9yEAAAAA:6aeIl_agLqFBsaViu4Audy7KdHJKtELPp6TF2KT0stCpng80APTfD7V1rZu0gpzqDqFp8ZKHy2E
review_title: Review

Meta-review: This paper explores the brain's activity in response to language, specifically targeting the signatures of syntax in the brain.  The authors specifically investigate the signatures of specific syntactic elements against the "typical" effort based syntax measures from some previous work.  

The title and abstract of the paper are clear and compelling, but the text of the paper muddies the message and this was expressed in the reviews.  There may be some debate in the literature as to if syntax and semantics are dissociable, and to what degree we can actually measure syntax in the brain, but I (and your reviewers) have trouble believing that any one actually thinks there is *no* syntax representations in the brain.  Certainly this is not a claim made by either the Federenko or Pylkkanen papers the authors cite. Federenko says "lexico-semantic and syntactic processing are deeply inter-connected and perhaps not separable" but doesn't claim that the brain doesn't "do" syntax. Pylkkanen says "Syntax in the brain is necessary to explain the fact that humans are exquisitely skilled at judging syntactic well-formedness, even for sentences that have no coherent meaning."

I suggest this paper either rephrase the arguments, more clearly articulate the issues they wish to address, or find another venue where the reviewers might be more read to debate *if* syntax is encoded in the brain.  That seems outside of the scope of ICLR.
Decision: Reject