Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The manuscript entitled “Can fMRI reveal the representation of syntactic structure in the brain” proposed a word embedding scheme with the consideration for incomplete sentence (used to test how future sentence structure is represented in the brain) and partial parses (used to represent the multiple top-down paths of the brain in interpreting language syntaxes). Words embedded by the proposed schemes are then mapped to the corresponding fMRI signals (ensured by the experiment design where each word is strictly 0.5 second) through ridge regression. Voxels in the brain that are significantly correlated to certain combination of the syntax structures are studied in the results.
main_review: Originality: Using a more comprehensive and integrated scheme to model complex syntax structure is the main contribution and novelty of this work. Various intrinsic feature of the syntax structure (e.g., predictability in incomplete sentence) could be captured by the proposed method, which is a major improvement over the previous works. 

Clarity: Method and result section of the proposed work is generally clearly written. With the help from the provided source code, the model should be partially replicable. Yet as the data and model are highly coupled in this work, it is difficult to fully test the model and evaluate it vs. other methods.
Section for InConTreGE (pp. 4, line 147-158) is confusing to understand, even with the help from Fig. 1: unlike a full sentence represented by a subtree, an incomplete subtree can have arbitrary number of forms (and the corresponding measurements such as node count). How exactly ConTreGE Incomp was constructed from this near-infinite composition of words? In addition, brain regions that are correlated with ConTreGE Incomp seemed too extensive and symmetric as in Fig. 4, which in contrast to common understanding of the high-order language processing unit in the brain (which might be true given the complex task involved in this study).

Significance: The proposed work would lead to superior representability for language-related functional neuroimaging analysis than traditional functional localization studies. In addition to the proposed paradigm, more specific language-processing researches can also utilize this framework for better modeling of functional neuroimaging data. The proposed work shall be significant to a highly specialized community (cognitive neuroscience in language study).
limitations_and_societal_impact: The author mentioned difference in the paradigm (naturalistic stimuli vs. designed sentences / phrases) would lead to different localization results in the discussion section, which is totally reasonable. However, they did not provide convincing premise to use the designated naturalistic paradigm: a more complex task for the subjects, such as the one adopted in this work, would introduce more external factors and confounding, which will possibly lead to inconsistent false positive regions being identified. The author is suggested to provide more neuroscientific evidence for the possible functional pathways during a reading task, in order to better elucidate the findings. 

A minor comment on the methodology: to the reviewer’s understanding, the ridge regression performed to map word features with fMRI signals is a 15*n->n model. The dimensionality of 15 might be too high for the ridge regression. There exist more powerful techniques to study the relationship between two sequences, where one of the sequences has higher dimensionality.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new set of syntactic tree embeddings, and uses them capture additional variance in fMRI data beyond simpler complexity metrics. I think the paper setup is compelling overall, it is well-situated relative to prior work, and the results are intriguing. My primary concern is that the paper doesn't conclusively establish that these new embeddings are purely syntactic and disentangled from either complexity metrics or (more ambitiously) anything that BERT can capture, and this damages the downstream claims about neural correlates of syntactic representations. 

I hope this can be addressed and solved in rebuttal/revision, since I find the paper good otherwise and hope to see it at the conference. 

EDITED: I think the authors have reasonably addressed my concerns, and am raising my rating accordingly. 
main_review: # Are these embeddings truly disentangled syntax embeddings? 
The paper makes strong claims that its embeddings need to be purely syntactic (both in the introduction starting l99 and the discussion starting l340). While much care is taken to ensure that semantics or other language features do not "leak" into the embedding explicitly via e.g. a lexicalized grammar, the paper does not provide an empirical demonstration of disentanglement / orthogonality. This is important in light of the paper's framing because the embeddings could nonetheless be "incidentally" correlated with other features, either by random chance or more realistically if there is unavoidable correlation between structures and semantics (e.g. longer sentences are used with some words/concepts but not others, which doesn't seem unreasonable). The paper should do more to demonstrate that these embeddings are truly "purely" syntactic (e.g. showing that on they are empirically orthogonal to the other embeddings), or moderate its claims. This is particularly important given that the distributed significance maps could equally well imply a distributed representation for syntax as a leak of other information into the embedding. 

To make room for this additional demonstration, I think the introduction containing extended background work could be shortened. Even if the extended background is kept, I would recommend writing a short introduction that puts the paper's contribution front-and-center, as it is currently well into the third page of the paper. A longer background discussion can follow. In addition, I think the section beginning l285, which seems to be a sanity check of the new embeddings' ability to capture  structural information without addressing the disentangling issue above, could be moved to the appendix if additional room is needed. 

# Other comments

- Surprisal may be unfamiliar to the broad NeurIPS audience, the paper may consider defining it if space permits. 
- I think the complicated acronyms like ConTreGE do not much to remind the reader about the content of the embedding, and are hard to match to the simpler acronyms like CC, CI, INC used in the figures. 
- The sliding window length for the embedding is first mentioned on line 188 but implicitly mentioned without definition on line 163. 
- Why do the semantic and syntactic feature dimensionalities need to be matched (l213)? 
- l214 has a stray x. 

limitations_and_societal_impact: The discussion of limitations is good, though missing the concern above regarding semantic leakage.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose new representational spaces that capture syntactical information in natural language above and beyond complexity-based metrics that have been typically used in language neuroscience. First, they build representations based on the constituency tree of a sentence as every word is processed by the parser. Then, they use a subgraph embedding algorithm to convert the subtree into a 15-D representation. To test several hypothesis about the type of syntactic information being encoded, the authors look at 3 different types of subtrees- the largest subtree completed when the current word is processes, the incomplete subtrees that can explain the sequence seen thus far under PSG production rules and the set of complete parses produced by a probabilistic model that are weighted by their probabilities. The 3 subspaces are compared against 4 complexity/load-based metrics and additionally, a semantic feature space derived from BERT. They go on to build encoding models for the different feature spaces and compare them through variance partitioning. Overall, the paper finds that the 3 graph-based spaces explain additional variance across the cortex as compared to traditional complexity-based features. Further, this effect is not localized to any region. Instead, it seems to be distributed across a network that is largely explained by semantic features, suggesting that syntax and semantic share neural substrates and there exists no isolated region for syntactic processing.

The study uses publicly released fMRI data from 9 subjects reading a chapter from Harry Potter .
main_review: Overall, this paper was well-written and easy to understand. While it would help to clarify how the 3 important ConTreGE features are computed for different constituents/nodes, I thought the idea of breaking down syntactic processing into 1) current subtree 2) anticipated subtree 3) all possible subtrees was thoughtful. Although the implications on language neuroscience are not novel (other work has shown or hinted that syntax and semantics likely have shared neural processing), to the best of my knowledge this is the first work demonstrating it in an encoding model setup with feature spaces that go beyond traditional effort-based metrics. 

Some clarifications & suggestions:
- Re ConTreGE vectors:
    - How many words in a sentence have different representations than POS Tags? For instance, all specifiers/heads would essentially have the same information as their POSTags but a complement or modifier might not right? It is unclear to me what the authors mean by 
> the largest subtree that is completed
For instance, why isn’t VP a valid subtree for _believed_ assuming that this verb has no complement? I understand that this is a secondary concern given that the representations are subgraph embeddings and thus, richer than the POSTags in many instances. But it is unclear to me how many times this is actually true and what are the rules that govern subtree selection. (My confusion was reinforced by the results of PD vs. CC in Fig. 3 & 4 + given that the random walk is still on nodes and not edges)
- Re ConTreGE Incomp vectors:
    - The example provided in Fig. 1C is used as a feature for _believed_ or _it_? I presumed _believed_ based on the hypothesis that the brain is already predicting future structure when it hears the word _believed_. It is unclear to me what this would look like for an already complete constituent, like _Harry_. Would this be any different from ConTReGE Comp? I agree that the idea of predictive coding (or so to speak) is interesting in terms of syntax, but do we really have the temporal resolution for this? Given that the word-space is downsampled and several of the features are very similar to the complete trees, I would like to know what the authors think.
- Re Incremental ConTreGE: How does this compare to say, using the softmax layer of a POSTagger? [I also think it would be more demonstrative to add examples of sub-trees where the structure itself is ambiguous: Maybe like a garden path sentence?]
- How do you deal with bidirectionally in BERT, given that the fMRI responses are generated causally?
- I had a hard time interpreting Fig. 3:
    - If the values are discrete from 0-5 , why not use a discrete & distinct colormap like viridis? It is not obvious to me what anything but dark green/white means.
    - Since the subjects were coregistered to a common space, I think this figure might be underselling performance. For instance, its possible that one subject has lower prediction overall due to SNR etc.. To this end, wouldn’t plotting just the number of subjects underestimate the unique variance explained by the new feature space?
    - What are the $R^2$ values? It seems that adding the ConTreGE features doesn’t improve performance substantially but this could be an incorrect reading of the figure. In addition to Fig. S5, I would appreciate a 3D model showing the unique variance explained for ever feature space that is added (and not the concatenation). Also, what is the significance threshold for the correlation values plotted in Fig. S5?
    - Missing reference that this is essentially variance partitioning [deHeer et al., 2017]
    - It seems strange to me that not enough voxels & subjects in the frontal cortex benefit from BERT when several studies (including the ones cited in this paper) have shown robust responses to semantic features in that part of the brain (especially given that none of the syntax models predict well there) Although, Fig. S5(k) suggests high correlation.
- My comments on ease of interpretation also apply to Fig. 4 wherein it would be helpful to show the percentage of significant voxels under each model as an individual bar. (I understand that it might be too crowded; but maybe stacked bars?) 
    - I find it interesting that comparable gains are found between BERT and some of the syntactical feature spaces in Broca’s area (contained in IFG) and sPMv (contained in MFG).
- How are the word features downsampled? Would be useful to add this to the paragraph starting in Line 229.
- For the _bootstrap_ test in lines 251-255: I am confused if the TR blocks were sampled with replacement like in a bootstrapping procedure or were randomly permuted like in a permutation test.
- What does the sliding window refer to in line 188?
- It would help to be more consistent in citing work that uses encoding models for fMRI. For instance, several publications that use encoding models for fMRI (JR King’s group, Jelle Zuidema’s group, Uri Hasson’s group, Alex Huth’s group) have been elided and there is inconsistency in what publications from the references are used at specific points (lines 63-64).
- Missing reference: Bhattasli et al., 2018 is closely related to _parser action count_ in Hale et al., 2018.

I am fairly certain who the authors are from having seen a previous version of the paper + a talk but the authors did not violate anonymity in any way.
limitations_and_societal_impact: The paper contains information on the limitations of the approach specifically but not broad societal impacts of encoding models and neuroimaging techniques.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: * The authors conduct a brain encoding study testing whether fine-grained information about the content of incremental syntactic representations contribute to predictions of neural activity, over and above measures of syntactic complexity.
* They design custom subgraph embedding representations to operationalize the idea of "syntactic representation." They find that these subgraph embeddings capture additional variance in brain activity above what is predicted by complexity metrics, and that this improvement in prediction is distributed throughout the language system.
main_review: The authors confront the important question of the representation of syntactic content in brain activity during language processing. I appreciate their creative choice of graph embeddings as a more specific, but still somewhat theory-free, proxy for mental representations of syntax. The paper could use improvements both in interpretation of results and in organization/clarity. But I believe the authors are making a good contribution to an important conversation. I mark the paper as marginally below acceptance threshold, but would change my review if the authors can (minimally) adequately address the theoretical concerns in the "Quality" section below.

## Quality

### Are ConTreGE embeddings distinctly syntactic?

There seem to be two distinct claims that are important to the authors, and which they take their results to support:

1. Syntactic representational content at a level deeper than complexity measures is present in recordings of brain activity. \*ConTreGE\* embeddings contain some of this representational content, and thus succeed at brain encoding over and above the predictions of complexity measures.
2. \*ConTreGE\* embeddings are distinctly syntactic, and so there exist brain regions selective for syntax iff there are regions well predicted by \*ConTreGE\* embeddings but not by BERT embeddings.

My hunch is that while \*ConTreGE\* embeddings do contain meaningful syntactic content, they also contain (correlates of) semantic content. This is because constituency trees still retain information at the syntax--semantics boundary. For example, in the PTB syntactic representation it is possible to read off semantic features such as number, aspect (from correlated tense features), and modality (from the presence/absence of auxiliaries).

Under this hunch, \#1 is valid but \#2 is not. Overlap in well-predicted regions between the \*ConTreGE\* embeddings and BERT embeddings (after controlling for ConTreGE success) may be due to the fact that the \*ConTreGE\* embeddings are not distinctly syntactic.

I would like the authors to address this question more directly. They do include (in lines 285--308) a test of the syntactic representational content of the derived ConTreGE embeddings, but this is of course not the same as demonstrating that the embeddings are distinctly syntactic. They might include more tests of representational content (or lack thereof) in the ConTreGE embeddings (e.g. visualizations of principal components of embedding space, decoding experiments, etc.), or post-hoc qualitative or quantitative evaluations of the differences between ConTreGE and BERT embeddings in real linguistic contexts.

As I understand it, this premise that ConTreGE embeddings are distinctly syntactic only matters for the inference about the overlap between syntactic and semantic content in neural representations. If there are other claims depending on this premise, I think they should also be reevaluated.

### Comparing the representations

I have several small qualms with interpretation of the differences between brain encoding with the different representations:

1. The relative success of the Incomp representation is taken as evidence that the brain is modeling future syntactic information (lines 281--283). But Incomp can contain more context than Comp in both directions. This is evident in the Comp--Incomp contrast in Figure 1B/1C: the Incomp representation carries information about content appearing before the largest completed subtree as well as after. Because of this, I don't think you can use the relative success of Incomp to argue for representations of future syntactic content.
2. Why do we believe that the Incomp representation should not represent uncertainty about the structure governing past content? (Why does Fig. 1C assume that "believed" is VBD? How is VBD chosen here over the alternative VBN as Fig. 1E makes apparent?)

## Clarity

The paper could benefit from some reorganization. The introduction is extremely heavy at the moment (a full two pages). I would recommend the authors compress this section and use some of the saved space to clearly outline the embedding methods and run intrinsic analyses / visualizations on these embeddings before moving on to brain encoding work.

## Other minor comments

1. I'm not a neuroscientist and can't address the validity of the encoding method used. I was slightly surprised to see the authors chose to learn a mapping from all the representations triggered in a TR, rather than using a convolution based on a hemodynamic response function.
2. Fig. 1e --- How is this a valid parse? If this is supposed to be a relative clause construction, the constituency structure is incorrect; the VBN should be a child of the NP. Also, an RC is implausible here with "Harry" -- you'd want a nonlexical NP such as "the person."
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper spurred a lot of discussion and back and forth with the authors, leading two reviewers to raise their scores. One reviewer thought the work is of limited interest and focused on a small community. I do not think this is grounds for reducing its score, since neuroscience is a main area in NeurIPS, definitely historically but also nowadays. 

A major issue was that the reviewers questioned whether the proposed embeddings are truly disentangled syntax embeddings. This point was partly addressed by the authors providing more analysis. However, the reviewers still would like to see a more careful and nuanced discussion of this issue, how to interpret the results in light of the syntax-semantics debate, and to make sure that limitations of the present work are discussed. 

The reviewers also mentioned problems with clarity and paper organizations -- specific questions were mostly answered, but paper organization should be improved in the next revision. 

Given the discussion, I believe the paper passes the bar for acceptance. My low confidence here is mostly because my insufficient familiarity with neurolinguistics. 
Decision: Accept (Poster)