Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is about two things: understanding learned NLP representations by finding and understanding similarities between *different* representations; and comparing these representations to fMRI data. The paper takes a new approach to both problems, and finds a dimension in the space of language representations that seems to correspond to something like "semantic depth of analysis." Furthermore, the comparison with fMRI data shows that many representations are correlated with brain activations.
main_review: This paper is very interesting methodologically, in the sense that it brings novel methods to an important problem, and also in the sense that these methods are highly worthy of further examination. I don't feel that I learned much about NLP representations or about brain representations, but I do feel like the approach taken here is going to be an important part of answering these questions in the future.

The paper is very clear, but there were still some parts that I had difficulty with. First of all, the methods in 3.3. It seems complicated: if the goal is to get a vector, for each type of representation, that encodes in some sense how similar another representation is, then why go to the trouble of the "tournament"? Why not just, for each element i, directly use the MSE for decoding the target representation using the representation i? And, for that matter, even in the tournament version, why not take into account the actual values of the MSEs rather than just the percentage for which the one was higher than the other? Given that there is a simpler idea at hand, I would guess that the tournament approach must be necessary. Otherwise the paper would have presumably taken a simpler approach. But I do not understand why. This leaves open the possibility that this approach was chosen empirically, which wouldn't be a good thing here. More explanation of these methods and whether and why they *must* be the way they are, is necessary.

More information would be welcome about what the "temporal transformation" is in 4.2.2.

In 4.2.3, the methodology is sufficiently complicated that I have a hard time following what exactly \rho is. More importantly, I have a hard time understanding why the \rho values should be sufficiently comparable to the representation embeddings that it would make any sense to project them onto the first basis vector of the MDS. I have a very strong feeling that it makes sense,but this part of the paper just goes too fast for me to understand exactly with these p vectors are.

Similarly, I have a hard time getting the intuitions behind the method in 4.2.4, and, as such, I don't feel completely at ease interpreting Figure 4. For one thing, what do the negative scores in the upper left subfigure mean?

An interesting but controversial property of this approach is that it tries to map the linguistic embeddings down to a fixed dimension using a linear mapping. This is interesting, but surely the only risk is not *underestimating* the relatedness between tasks. Imagine in the worst case that all information is lost in the mapping, and we always get the zero vector, for two different representations. Surely in that case we should have great performance decoding and thereby *overestimate* the relatedness. 

I'll return to my remark that I'm not sure I learn a lot empirically - although, I stress, this paper does have a lot to recommend it, particularly methodologically. I don't really see much in the brain map in Figure 3. The text tells me there's something - not necessarily much - and I guess I trust that, but it does not seem like there's much there. And I have a hard time getting *anything* useful out of Figure 4, aside from the fact that these representations predict voxels. If there is more, and more interpretable, content to be had in these results, I'd encourage the authors to draw it out more. Similarly, the similarity matrix in Figure 2 isn't all that informative, although the MDS is certainly intriguing. However, it leaves a lot of open questions. Is this first component really "semantic depth"? What further tests could we do to probe whether that's the case? How could we operationalize that notion? Second, what is this second dimension doing? And, while the paper suggests that the later layers of some models come close to the next word embedding, I don't see them (visually) coming very close, and a glimpse at the similarity matrix in Figure 2 confirms that for me. The next word embedding is really far from everything. So... doesn't that mean these language models are bad?

In short, I think there is a lot of promising exploration made possible by this paper. I'm not sure there is enough exploration and discussion, as per my comments. And I find the methodology either too complicated, too sparsely motivated, or just hard to follow because it's too dense. 

Comments in author review period
===========================

My overall assessment of this paper is exactly the same, except that I realize that the methods (based on Zamir) are slightly less original than I had thought. I am lowering my score to a 7 to reflect this. It is still a very interesting if preliminary exploration.
limitations_and_societal_impact: Yes, though see above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Using a slightly modified (but relatively straightforward version) of an analysis that’s recently been called ‘neural taskonomy’, this work attempts to use the relationships (read: transfer learning affinity) among different natural language representations to assess the organization of linguistic representation in human fMRI data via regularized linear encoding models.  The authors first contribute a modified version of a previously used encoder-decoder framework to create a 'task' (linguistic representation) affinity matrix that demonstrates conspicuous, interpretable structure when reduced via MDS to fewer projections. They then use the structure discernible in this matrix to ascertain whether similar structure (specifically, a hierarchy of representation from word embeddings to later, deep transformer layers) is evident in the brain. 
main_review: The ‘taskonomy’ portion of this paper (in that it closely mirrors the work of Zamir & Colleagues) is mostly clear and well-constructed; the selection of models seems comprehensive enough (spanning relatively simple word embeddings to more complex transformers), and the range of linguistic tasks covered seems (at first glance) sufficient to recapitulate the diversity of semantic and syntactical structures that should presumably define the neural code. Some minor points of clarification I think could benefit this section are:
- Clarifying what exactly is meant by the ‘mean squared error’ on some % of the data in the generation of the pairwise tournament matrix (135-139). How is the data being divided? What exactly is the data in this case? A particular set of sentences in the corpus being tested? This section – though perhaps routine in principle – left me utterly complexed.
-Reformulating the sentence starting at line 161 (“We also included the GloVe 162 embedding for the next word in the sequence, which constitutes a low-dimensional representation the ideal output of a language model”) to both remove the typo (assuming it’s missing a colon) and to clarify why a low-dimensional representation is the ideal output of a language model. This is not immediately clear to me a priori, and I might even be inclined to say the opposite, depending on the functional context in which the representation is leveraged.
-Is the vertical (top to bottom) dimension in the MDS interpretable? The EFA suggests the second most significant factor still accounts for 10% of the variance. Does this 10% come with interpretability? If not, is it really the case that the low-dimensional structure among the natural language representations is interpretable in only one direction?
-What is so different about the GLOVE Next Word embeddings? Why their consignment to the periphery of the MDS?

All in all, I find this section particularly compelling, and am intrigued by the structure the authors seemed to have uncovered – though an attempt at interpreting other dimensions after the 1st could make it just a bit more satisfying.

The ‘neural’ portion of this paper is the less compelling of the two – not necessarily due to methodological flaws, but to a lingering sense that some of the author’s central claims (including, most importantly, the recapitulation of the taskonomic representational hierarchy across the brain) are not so neatly supported by the data. Take for example the BrainView in Figure 3. The authors make a significant point of underscoring a hierarchy of representation (from word embeddings to late transformer layers), but most of the areas activated by the ‘later’ (presumably more complex) representations are early perceptual areas (including most conspicuously, early visual cortex). Anterior inferotemporal cortex (the main site of object recognition, and presumably the location where vision interfaces with language) is dominated by ‘earlier’ (presumably less complex) representations. I can’t quite make sense of this in a narrative about representational hierarchy – but this brings me to a second point: the authors do not make clear what we should be looking for. Are there known representational hierarchies in the neuroimaging of language? If so, how are they characterized and what are their stereotypical motifs? Obviously, the specific loci of syntactic and semantic structures in the brain remain a matter of significant (even hot) debate – but some reference point here (from Caramazza, Federenko or other similar researchers) could help to situate and better direct the reader’s attention to key data points of high interpretive value. 

Similarly, the lack of an overall gestalt in Figure 4 (at least a gestalt comparable to that provided by the MDS plot in Figure 2) gives the impression that the authors relied a bit too heavily on a data-driven approach without a clear sense of the theoretical insights to be gained from their otherwise powerful methodologies. What does it mean that the GPT models were overall most predictive of neural activity? Is there discernible structure in the MDS computed on the discriminability score matrix (perhaps structure comparable to that found for transfer affinities)? For that matter, what do all of these answers look like in parts of the brain localized or functionally segmented with a language-specific task? Without answers to these questions, I find it difficult to say what I’ve learned from the ‘neural’ half of this ‘neural taskonomy’ analysis. 

I think this paper has significant potential, and would be happy to add to my score if the authors can more clearly address the following two points in their rebuttals:
- Is there a distinct linguistic hierarchy in (either the whole or a functionally localized linguistic portion of) the brain? If yes, how is it defined, where exactly is it located and does the hierarchy unveiled by your task affinities match this hierarchy?
- What is the overall takeaway from the rankings of the 100 representations in terms of their power as predictors in neural encoding models? 

limitations_and_societal_impact: Yes -- though perhaps some thoughts on the limitations of purely data-driven approaches to computational neuroscience would have been apropos.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The submission proposes to study similarities between language representations in different language models. First, a similarity measure between language representations is derived. Then, the matrix of similarities is reduced to two dimensions using MDS. The authors then discuss qualitatively this low-dimensional space of language representations. Then, linear encoding models of fMRI recordings are used to project the first MDS dimension on the cortical surface of a subject listening to narratives. Finally, the matrix of similarities is used to predict the performances of linear encoding models.

main_review: *Originality:* Comparing representations from intermediate layers of deep neural networks is an interesting topic and has received a lot of interest in the recent years, as adequately discussed in the introduction. This submission proposes a new method to compare representations, using a procedure based on bottle-necked encoder-decoder across representations, creating tournament matrices from pairwise comparisons of representations, and stacking the tournament matrices first eigenvectors to create a similarity matrix. This procedure seems partially novel and partially inspired from Zamir et al [38], although it is not clear how this work exactly relates to previous contributions. In particular, there is no quantitative nor qualitative comparison with what would be obtained with CCA or CKA. 

*Quality:* The submission is technically sound, although the motivation of each step of the procedure could be better explained. The results look reasonable and robust, and adequate care about generalization and overfitting is mentioned. The authors note the limitation of centering their similarity measure around the GloVe word embedding. The shared code and data is appreciated, although not all code files reach a reasonable level of documentation. Importantly however, the authors do not compare their procedure to other similarity of representations methods .

*Clarity:* The paper is clearly written and easy to follow, except a few convoluted sentences (e.g. L36, L74). However, the paper could better inform the reader about what parts of the method to build the similarity matrix is novel. The discriminability scores in section 4.2.4 are not enough motivated and explained. In particular, it is not clear why "high discriminability scores (red) [indicates] that the representation embeddings broadly capture the structure of the brain’s language representations". 

*Significance:* It is hard to judge the significance of the method section (section 3), since no comparison with previous work is proposed. However, using MDS on the similarity matrix and the corresponding discussion around Figure 2 is interesting. The projection over the cortical surface is not surprising, as the first MDS dimension is mainly a measure of the DNN layer, and similar gradients have been described over encoding models built from DNN layers [e.g. Kell et al, Neuron 2018].

*remark:* NLP is not defined.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: * The authors propose "representation embeddings" (vectors describing transfer performance from task i to task j) as a low-dimensional space describing the content of language representations.
* They analyze the derived representation embedding space and claim that the main principal component tracks "depth" (line 192) of linguistic processing.
* They conduct a brain encoding task and evaluate 100 models on this task. They claim that a model's brain encoding performance tracks with its behavior in this low-dimensional representation embedding space, and that this low-dimensional behavior also maps intuitively onto the spatial representation of language in the brain.
main_review: I think the paper is exceptionally well motivated among brain encoding papers, and clearly written. However, I think the results and conclusions fall short of this motivation. I think the results are marginally worth publishing as-is, but the paper could be strengthened by addressing some of the conceptual issues below.  

  
### Quality/significance
  1. I strongly support the authors' mission to move past coarse, qualitative labels such as "syntax," "semantics," and "high/low-level" in the quest to understand both neural network and brain representations of language. But unfortunately the conclusions of the paper still seem to entirely depend on these pretheoretical notions. The MDS analysis is mainly framed as interesting because its first principal component seems to recapitulate the syntax--semantics processing trajectory (lines 183--187). The brain encoding analysis is also framed as a success because it matches up with existing notions of the spatial distribution of "higher-order" processing in the brain (line 266).As such, I think the original motivation of the authors is not fully accomplished in this paper. Ideally we would be able to use novel quantitative tools to go beyond our current conceptual framework (e.g. proving their worth in exploratory data analyses, or making predictions at a granularity / in a domain not afforded by the original qualitative distinctions).Suggestion for improvement: Explain concretely in your 'future work' section how this paper (or the next paper) can go beyond the qualitative distinctions of "syntax," "high/low-level," etc. 
  2. The neuroscientific contribution is not clear to me. The authors claim to test whether the "MDS dimension could capture patterns of hierarchical processing observed in cortex," and claim that the resulting voxel map of Figure 3 answers in the affirmative. But without a more specific (quantitative / statistical?) statement of the claim and without any reference / comparison to alternative hypotheses, it's not clear what success or failure should look like for this evaluation.Suggestion for improvement: Evaluate against a quantitative baseline performance measure, or design (post-hoc?) statistical tests to make concrete some of your localization claims.
  3. I admire the creative use of higher-order "representation embeddings" here, and the mapping of its principal components back onto higher-order features of brain encoding models. But I have trouble seeing a path forward for the cognitive neuroscientific project here. This analysis setup seems to only be capable of mapping between-model representational distinctions (e.g. shallow vs deep linguistic processing) brain areas. The high-level analysis erases by design any between-stimulus distinctions; as such, it blocks us from exploring more advanced questions about fine-grained distinctions in linguistic representational content (as random examples -- how/when/where is implicature computed, how/when/where are scope ambiguities resolved, etc. etc.). Because the authors seem to be suggesting this sort of analysis as the beginning of a broader research program, I would appreciate if they could address the limits of the questions they can answer, in terms of representational content.Suggestion for improvement: Add discussion to conclusion.

### Clarity
  1. I had a very difficult time understanding both the motivation for and the mechanics of the discriminability analysis of section 4.2.4\. I don't understand what this analysis contributes beyond the more basic encoding results in Fig. 4 lower right. Why is this measure computed on pairs of representations? And why is a high discriminability evidence of "broadly captur\[ing\] the structure of the brain's language representations," over and above the basic correlation values?Suggestion for improvement: I would definitely appreciate more motivation of the particular method, which is currently very difficult to understand. (It's a second-order analysis of brain encoding models trained on second-order language representations!)
  2. Minor comments/questions
    * Line 261: What is the motivation for z-scoring these values?
    * Section 4.2.4, M equation: Why are you computing correlations here? My understanding is that both $\hat{h}$ and $\rho$ are nonrandom scalar values.
limitations_and_societal_impact: See above for scientific limitations. No issues re: societal impact.
review_title: null

Meta-review: Reviewers found that the method presented in this submission is useful and casts our understanding of language processing in the brain in a new light.

I encourage the authors to update their manuscript to include the results from the discussion with the reviewers, as these were decisive.

Decision: Accept (Poster)