Low-dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
Authors: Richard Antonello
Javier S. Turek
Vy A. Vo
Alexander Huth
Venue: NeurIPS 2021 Poster (2021)
Keywords: fMRI Encoding Models
Language Representations
Natural Language Processing

Abstract: How related are the representations learned by neural language models, translation models, and language tagging tasks? 
We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks.
This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.

Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
Richard Antonello
Javier Turek Intel Labs javier turek@intel.com
UT Austin
rj mjantonello@utexas.cc
Vy Vo
Alexander Huth
Intel Labs
UT Austin huth@cs. utexas.ecccm edu
vy.vo@intel.co com
Abstract
How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision investigate the structure among 100 different feature spaces extracted from hidden represen- tations of various networks trained language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language represen- tation embedding because it encodes the relationships between representations needed to process language for a variety of NL.P (natural language processing) tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using IMRI. Additionally, be used to create a metric which highlights the brain's 's natural language processing hierarchy, This suggests that the embedding captures some part of the brain's 's natural language representation structure.
find that the principal dimension of this structure can
Introduction
There are multitude of common techniques for analytically representing the information contained in natural language. At the word level, language is often represented by word embeddings, which capture some aspects of word meaning using word co-occurrence statistics [8] 25 Language representations
that highlight specific linguistic properties, such as parts-of-speed 29 or sentence chunks ([), or that utilize well-known NLP models such the intermediate layers of pretrained language models 66, 10, 21] (7), are also frequently studied [12 [T]. fields such as linguistics, natural language processing, and cognitive neuroscience, qualitative adjectives representations - e.g. "low-level" or "high-level" and "syntactic" or "semantic". The use these words belies unstated hypothesis about the nature of the space of language representations namely that this space is fundamentally fow-dimensional, and therefore that the information from the representations in this space can be efficiently described using few categorical descriptors. In this work, we attempt to directly map the low-dimensional space of language representations by generating "representation embeddings" using a method inspired by the work of Zamir al. [44] This method uses the transfer properties between representations to map their relationships.
often used to describe these language
The work described here has two main contributions. First, used the representation embeddings to demonstrate the existence low-dimensional structure within the space language representations.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We then used this structure to explore the relationships between and gain deeper insight about frequently used language representations. How do the intermediate layers of prominent language models relate to one another? How do the abstractions used by these layers evolve from low-level word embeddings of a context to a representation of the predicted next word for that context? Do different language models follow similar representation patterns? What are the differences between how unidirectional and bidirectional language models represent information? These are examples of the types of questions we can explore utilizing our representation embedding space, Second, we showed that this low-dimensional structure is reflected in brain responses predicted by these representation embeddings. In particular, show that mapping the principal dimension of the representation embeddings onto the brain recovers, broadly, known language processing hierarchies. We also show that the representation embeddings can be used to predict which representations map well to each area in the brain.
2 Related Work
Our work closely follows the methods developed by Zamir et al. 44 Those authors generated a laxonomy of vision tasks and analyzed their relationships in that space. This was achieved by applying transfer learning across vision tasks, and showed that this can reduce data labeling requirements. We used some of their methods, such as transfer modeling and normalization using analytic hierarchy process (AHP) [28]. for our analysis on language tasks. Following the same lines but for NLP, the work from Vu et al. [37 explores transferability across NLP tasks. Those authors focus on the application of transfer learning to show that the amount of labeled data, source and target tasks, and domain similarity are all relevant factors for the quality of transferability. Kornblith et al. (20) investigated and compared techniques for measuring the similarity of neural network representations, such canonical correlation analysis (CCA) and centered kernel alignment (CKA).
Our analysis of the relationship of between the language representation space and the brain leans heavily on the concept of "encoding models". Recently, encoding models have been widely applied as predictive models of brain responses to natural language [1 40). Jain and Huth [88] applied encoding models to representations of an LSTM language model that incorporates context into neural representations. These representations are explored for different timescales of language processing in Jain et al. [99. Toneva and Wehbe (33 explored the encoding model performance of different language model layers to improve neural network performance. Caucheteux and King (5) observed differences in the convergence of hidden state representations from ANNs trained on various visual and language processing tasks to brain-like representations. Schrimpf et al. 30 systematically examined the performance of encoding models across variety of representations and neural response datasets and observed consistent high performance of Transformer-based model hidden states at predicting neural responses to language. Wang al. [39] constructed encoding models on a set of 21 vision tasks from Zamir et al. (44) with the objective of localizing tasks to specific brain regions. Those authors built a task graph of brain regions showing similar representation for highly transferable tasks.
3 Methods Overview
Our method aims to use the transfer properties between different language representations to generate quantitative descriptions of their relationships. These descriptions can then be used an embedding space without labelling any individual representation with predefined notions of what that repre- sentation is or how it relates other representations. This is analogous how word embeddings use co-occurrence statistics to generate quantitative descriptions of each word without labelling any individual word with our predefined notions of what that word means. By using transfer properties to generate the embedding for each representation, we implicitly assume that language representations with similar transfer properties are themselves similar.
We define a representation & as a function over a language input that extracts a vector of meaningful features for each word. Some representations can be computed independently for each word, such as word embeddings [ 10 25), while others depend on both a word and its context, such as part of
speech labels [1] the hidden state in some intermediate layer of language model [5 6 27 30). We define a stimulus s as a string of words drawn from some natural language source, and S the set of all stimuli from that source. To generate a representation embedding over set of representations T that are derivable from the same stimuli S we will construct time-dependent mappings between

---

representations. For a representation t E T, and for all times in our stimulus E S, the vector t(s;) defined. Next, for each pair of representations 1, This provides transfer mapping / (.) that can be learned for each pair of representations, fi- (s,)) (2,))) Vj.
will attempt to map ti (s,) to (s,).
However, learning these transfer mappings directly would make comparison across representations difficult, as many of the representations have different dimensionalities. Instead, we adapted the encoder-decoder framework proposed in Zamir et [44, which ensures that every transfer mapping has the same input dimensionality. This method proceeds in three parts (Figure I);
First, we train a linear encoder for each representation that compresses the information in the input down into a latent space. The latent space should only contain information necessary to predict the given representation's 's features from input. This is accomplished by training encoder-decoder from the input to the given representation via a latent space, then discarding the decoder.
Second, we train new decoder for each pair of representations that uses the previously generated latent space for one representation predict the other representation.
Third, we evaluate the performance of each decoder targeting a given representation relative to every other decoder targeting that same representation. This provides measure of how much of the information in one representation is present in every other representation. Just as words that have similar co-occurrence statistics-we wween measured against a large enough group of other words are thought to have similar meanings, we argue that pairs of representations that have similar transfer properties properties-wher wwen measured against large enough group of other representations- have similar information content.
e.g. GPT-2 S layer
Lt
Di1
(S)
Fin
D1
S
GloVe
U(S)
Eus
D
Lrz
D D12
(2(
e.g. PoS
The encoder-decoder strategy used in method, adapted from Zamir al. 44 S is the natural language stimuli. We chose to represent stimuli in the universal input feature space U(S) as GloVe Ve word embeddings. Encoders Es, were trained using bottlenecked linear encoder-decoder network, which outputs to (S) (blue arrows). The decoding half of this network was then discarded, and the encoding half used to generate latent space L., for each representation 4, Then, decoder Dr, is trained from each latent space to each representation j (orange arrows). The performance of decoders that map the final representation are then compared to another.
Figure 1:
3.1 Generating Representation Encoders
We define (S) the universal input feature space for our stimuli S. In practice, this space should represent the input with high fidelity, so could be selected as a one-hot encoding of each input token, word embeddings for each input token. We use Glo Ve word embeddings for U(S). Now let T be a set of representations over S. For each representation E T,, we generate an encoder Ex(-) such that the encoder extracts only information in USS) that is needed predict I(S). We do this by using a bottlenecked linear neural network that maps every u E US) to an intermediate low-dimensional (S)) and then maps the given representation space,
latent space E.- (U((S))
t(s) = ((E(
Vs E SEAAU U(s),
where f(-) is mapping from C, to f(S). We used a small latent space of 20 dimensions to encourage the encoder to extract only the information in US) that is relevant to compute (SS). Experimentation showed that variations in the latent space size do not meaningfully change encoder performance.
We assume that elements j in string sy are properly treated, such that can exclude the positional indexes for all s. Thus, define U(S) (U(s) 8) as the result of applying function ((-) all elements S.
3

---

Once we have learned these mappings, we assign E, to be our representation encoder for each representation & T. Regardless of the dimensionality of the representation, each latent space has a fixed dimensionality, which enables a fair comparison between representations.
3.2 Generating Representation Decoders
The encoders for each representation generate a latent space Le that extracts the information in (S) relevant to computing t(S), while compressing away irrelevant information. For every pair of representations (f1, 12) E T, we next generate decoder Duy-tz such that Di,-to (Lt,) Di, E, (U)) approximates (S)) This yields a total of n² decoders, where n = [T] the total number of representations. All networks were trained with batches of size 1024 and standard stochastic gradient descent with learning rate of 10 14-4 for the initial encoders and 2 10 10-5 for the decoders. Hyperparameters were chosen via coordinate descent.
3.3 Generating the Representation Embedding Matrix
Finally, we compare the performance of the decoders for each representation in order to generate the representation embedding space. To ensure that comparisons are made on equal footing, we only compare decoders with the same output representation (e.g. Dt, is compared with D Dts-sta- Dig-tt,, etc.). This critical, because evaluating representations with varying dimensionalities would unfairly bias the results. This and related challenges were explored in detail in Zamir et al. [4].
We used the decoders generate pairwise tournament matrix W. for each representation by "fighting" all pairs of decoders that output to representation & using held-out test set Steet of sentences. Element (i,j) in W, contains the ratio of samples in the test for which D has
lower MSE than Des- i.e., Wt FacS Dr. (8)> (1)
For example, if the decoder
[ss]
DA-c has lower mean squared error than decoder DB-Cc for 75% of the data in Stest: we assign the ratio of 0..55
to entry (A, B) in the tournament matrix We for representation C.
We then use this set of pairwise comparisons to approximate a total order over the quality of all encoders for each decoded representation. This was done using the Analytic Hierarchy Process (AHP) (28). a technique commonly used in operations research to convert many pairwise comparisons into an estimated total order. AHP establishes that the elements of the principal eigenvector of W constitutes good candidate for a total order over the encoders for f. This eigenvector is proportional to the time that an infinite length random walk on the weighted bidirected graph induced by W, will spend at any given representation. Thus, if the encoder for one representation is better than the others, the weight on that representation will be higher as compared to the weights of other encoders. This eigenvector is then normalized to sum to 1. This procedure yields a length n vector for each of the n representations. We set a value of 0,1 into each vector at the position corresponding to the target representation, and then stack the resulting length vectors together into the representation embedding matrix, R.
4 Results
4.1
Language Representation Embeddings
We applied our method to a set of 100 language representations taken from precomputed word embeddings pretrained neural networks for NLP tasks operating on English language inputs We extracted representations from numerous different tasks, which included three word embedding spaces (Glo Ve, BERT-E, FLAIR) (25
1. three unidirectional language models (GPT-2 Small, 10
6, 41 two masked bidirectional language models (BERT, ALBERT) 10 21), four common interpretable language tagging tasks (named entity recognition, part- of-speech identification, sentence chunking. frame semantic parsing) 1. and two machine translation models (English- + Mandarin, English - German) [32]. We also included the Glo embedding for the next word in the sequence. which constitutes a low-dimensional representation of the ideal output of language model. For multilayered networks (language and translation models), we extracted all intermediate layers as separate representations. Full descriptions of all representations, as well as details on how each feature space extracted, can be found Appendix A
GPT-2 Medium, Transformer-XLl
27
Data and code for this paper are available htttp:///gicbbb..c////hhhbbbbbb
4

---

Space
PO5
FRAME
GPT-2 Nedium
GPT-2
English Oarman
Decoddd Feature Space
Mandarin
FLAIR
Trandfomert
BEAT
GloVe
10
GloVe Noxt Word
3.05 5.10
Nummer of factire
Figure 2: Language Representation Embeddings with Low Dimensionality: (Left): The representation embedding matrix R shows how well a given linguistic feature space (encoder, columns) transfers to another feature space (decoder, rows). For better visualization, rows and columns corresponding to different layers from the same network have been scaled down in this plot. A full-scale matrix is in the supplementary material. (Right): Applying multi-dimensional scaling to the representation embedding matrix reveals oow-dimensional structure in the linguistic feature spaces. It is dominated by a left-to-right progression from the input word embedding, syntactic and semantic tagging tasks near the middle layers of language models. to the next word embedding. Multidimensional scaling was weighted such that each full model had equal weight, ensuring that language models were not more influential on account of having more layers. The dominant main diagonal was set to 1.1 to preserve the effects off-diagonal values. The scree plot in the lower left shows that these first two dimensions explain substantially more variance (22% and 10%) ) than other dimensions, demonstrating that the structure in this space is low-dimensional.
Applying our method these representations, we generated representation embedding space (Figure 2}. Generating this space took roughly one week of compute time on a cluster of 53 GPUs (a combination of Titan X, Titan V. and Quadro RTXs) and 64 CPU servers of different characteristics (Inte Broadwell, Skylake, and Cascade Lake). The representations were trained on a text corpus of stories from The Moth Radio Hour [36] which totalled approximately 54,000 words. The held-out test set for the tournament matrix included a story of 1,839 words. Each row shows the relative performance of each of the 99 other representations in decoding a given representation. Each column shows how well a given encoded representation can decode to the other representations. Note that there is a distinct asymmetry: if given representation (e.g. part-of-speech tagging) is never the best encoder for any other representation, then its column will contain very low values. However, other representations may decode to this one quite well, so the corresponding part-of-speed row can contain higher values.
4.1.1 Multidimensional Scaling Analysis
To better visualize the relationships between the representations, we performed multidimensional scaling (MDS) 34 on the rows of the representation embedding matrix. We then plotted the representations according to their locations on the first two MDS dimensions (Figure [ right). This showed that the layers of each of the unidirectional language models form similar trajectories through the space, beginning near the word embeddings (green) and ending relatively close to the word embedding (black). This trajectory is monotonic on the first MDS dimension, but values on the second MDS dimension rise and then fall, with the highest values assigned to the middle layers (e.g. layer of 12 in GPT-2 small). Language tagging tasks that require semantic information,
'Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
5

---

such as framing and named entity recognition (blue), are close in space to the middle layers of the unidirectional language models. Language tagging tasks that required syntactic information, such as part-of-speech identification and sentence chunking (red), are similarly clustered.
We also observe several other interesting relationships. Like language models, the machine translation representations also begin near the word embeddings. However, translation to Mandarin (Eng-Zhh appears align more closely to the unidirectional language models than translation from English to German. This may be because English and Mandarin are less closely related to one another, and could thus require deeper syntactic and semantic processing for successful translation.
We also observe that the bidirectional language model representations begin farther away from the input word embedding, closer the middle of the unidirectional language models. However, they also end up at a similar point, closer to the next-word embedding. This may be a result of the procedure to extract a feature space from the intermediate layers of the bidirectional language models, which differed meaningfully from extracting the intermediate layers of the unidirectional language models. Since these networks are trained on a masked language modeling task, we extracted the hidden representation for the token immediately preceding the mask token.
To study the dimensionality of the space of language representations, we examined the variance explained by low- diweenisiionll approximations of the embedding matrix. The scree plot in Figure 2 shows the result of applying exploratory factor analysis (EFA) [43] to the representation embedding matrix. We see that the first few dimensions explain much more variance in the matrix than the other dimensions. The first dimension alone constitutes 22% of the total variance. This demonstrates that the representation embedding matrix-an and thus, we infer, the space of language representations low-dimensional structure.
4.2 Representation Embeddings Predict fMRI Encoding Model Performance
Next, we hypothesized that these language representation embeddings are useful representation of how natural language is processed in the human brain. Since it is the only system truly capable of producing and understanding complex language, the human brain is an excellent litmus test to determine whether this www-dimensional embedding captures the underlying structure of linguistic information. To test this hypothesis, we used the representation embeddings generated above to predict how well each representation is able to predict fMRI data collected from the human brain as the subject is listening to natural language stories.
4.2.1 fMRI Data
We used functional magnetic resonance imaging (fMRI) data collected from 5 human subjects as they listened to English language podcast stories over Sensimetrics S14 headphones. Subjects were not asked to make any responses, but simply to listen attentively the stories. For encoding model training, each subject listened to at approximately 5 hours of unique stories across 5 scanning sessions, yielding total of 9,189 datapoints for each voxel across the whole brain. For model testing, the subjects listened to the same test story once in each session (i.e. 5 times). These responses were then averaged across repetitions. Training and test stimuli are listed in Appendix B.. Functional signal-to-noise ratios in each voxel were computed using the mean-explainable variance method from Nishimoto et al. [24] on the repeated test data. Only voxels within 8 mm of the mid-cortical surface were analyzed, yielding roughly 90,000 voxels per subject.
MRI data were collected on 3T Siemens Skyra scanner at the University of Texas Austin Biomedical Imaging Center using a 64-channel Siemens volume coil. Functional scans were collected using gradient echo EPI sequence with repetition time (TR) 2.00 s, echo time (TE) = =0.88 ms, flip angle = 71", multi-band factor (simultaneous multi-slice) = 2, voxel size = 2.6mm x 2.6mm x 2.6mm (slice thickness 2.6mm), matrix size 84x84, and field of view 220 mm. Anatomical data were collected using a TI-weighted multi-echo MP-RAGE sequence with voxel size = 1mm x Imm x 1mm following the Freesurfer 15
morphometry protocol.
Experiments were approved by the University of Texas at Austin IRB. All subjects gave written informed consent. Subjects were compensated for their time at rate of $25 per hour, or $262 for the entire experiment. Compensation for the subjects totaled $1260.

---

Group Average per Anatomical ROI
1st MDS Dimension
Left Lateral View
Right Lateral View
Single Subject
Left Medial View
Right Medial View
Figure 3: Embedding Brain Voxels in the first MDS dimension: Projection of the encoding perfor- mance vectors for each voxel in one subject (lower center flatmap and all 3D views) and averaged over all subjects within anatomical regions (upper center flatmap) over the 100 representations onto the first MDS dimension of the representation embeddings, which explains about 20% of the variance in the representation embeddings. Voxels with high values in this embedding (red) are better explained by representations that are more positive on the main MDS dimension (e.g. later language model layers), and voxels with low values (blue) are better explained by representations that are more negative (e.g. word embeddings). This dimension notable as it is the main dimension along which language representations evolve from "carlier" representations such as word embeddings, to "later" representations such as intermediate layers in deep language models. Anatomical ROIs were defined automatically in each subject using Freesurfer with the Destrieux 2009 atlas [9. Similar maps for the other subjects and a plot showing the numerical projection of regions in the MDS space are shown in Appendix D.I.
4.2.2 fMRI Encoding Models
Encoding models predict a measured brain response B, e.g. blood-oxygen-tevel dependent (BOLD) responses recorded with IMRI, based on the stimulus observed by the subject. Due limitations on dataset size, encoding models are typically structured as linearized regression, where each regression predictor is some feature of the stimulus. Encoding models can provide insight into where and how information is represented in the brain. For instance, an encoding model that has a feature for whether given auditory stimulus a person's 's name can used determine which regions of the brain are activated by hearing names.
Here constructed voxelwise encoding models using ridge regression for each of the 100 language representations! analyzed above. Let g(t;) indicate a linearized ridge regression model that uses a temporally transformed version of the representation f; as predictors. The temporal transformation
accounts for the lag in the hemodynamic response function 223 17]. We use time delays of 2, 4, 6, and 8 seconds of the representation generate this temporal transformation. For each subject æ, voxel v. and representation t. we fit a separate encoding model to predict the BOLD response B, i.e. B(z,,1,,) (......)) (t4). An optimal ridge parameter a was estimated for each I(,. by using 50-fold Monte Carlo cross-vale validatio,,, with a held-out validation set of 20% of the datapoints from the training dataset. We then measured encoding model performance p by computing the correlation between the true and predicted BOLD responses for separate test dataset consisting of one story. That is, P(a.v,t,) measures the capacity of representation 4 to explain the responses of voxel e in subject
7

---

4.2.3 Mapping the First Dimension of the Representation Embedding Space to the Brain
If a brain area was specialized for processing a specific type of information, then we would expect representations that capture that information to be good predictors of that brain area. Each voxel in the brain can thus be thought of as another language representation, and we can infer where that representation would lie the MDS space by projecting it onto those dimensions.
The main MDS dimension (left to right in Figure 2 is especially interesting as it seems to capture an intuitive notion of language representation hierarchy. Representations with low values along the main dimension include word embeddings as well as the earliest layers of most of the language models and machine translation models. Representations with high values along the main dimension include the deeper layers of these models, as well as the majority of the interpretable syntactic and semantic representations. We tested whether this MDS dimension could capture patterns of hierarchical processing observed in cortex. First we defined P(x,e) = (store(()ta...4) ... P(z,,,t..))) as the 100-dimensional vector that denotes the encoding model performances for each subject and voxel v, z-scored over all representations T. We then projected each P(z,,) onto the first MDS
dimension by taking their dot product. This gave us a value that quantifies which part of the first MDS dimension best predicts the activity of a given voxel.
Figure D shows shows the projection of each voxel for one subject (lower center) and averaged across subjects in each anatomical ROI (upper center) onto the first MDS dimension. Blue voxels and regions are better predicted by representations that are low on the first MDS dimension, whereas red voxels and regions are better predicted by representations that are high on the first MDS dimension.
While the literature has not settled on single hierarchical view of language processing in human cortex, one point of general agreement among existing theories is that there is a set of "lower" language areas, including Wernicke's 's area/auditorry cortex (AC), Broca's area (BA), and the premotor speech area sPMv [1 16]. Across our five subjects, we see negative projections on the first PC (corresponding earlier LM layers) in AC, but both positive and negative projections in BA and sPMv. This matches other results using narrative stories and encoding models Z where it was found that, of these three core language areas, AC is the best explained by lower-level features (phonemes and sound spectrum).
Outside of the core language areas, the literature is more divided on representational hierarchy. It is broadly agreed upon that these other areas, including much of the temporal, parietal, and prefrontal cortex, constitute the "semantic system" [4) in which language meaning is derived and represented. Our data show that most of these regions have positive projections on the first PC, corresponding to later LM layers. More fine-grained analyses have suggested that some areas, such as the angular
gyrus (AG) and precuneus (PrCu), contain the highest-level representations [9 [22.. This matches
our group-level results, which show that the AG and PrCu have the most positive projections on the first PC of any brain areas.
4.2.4 Predicting Encoding Model Performance with Representation Embeddings
If the representation embeddings reflect how the brain represents linguistic information, then it should be possible match each representation embedding ry its corresponding performance vector P(),) which describes how well that representation can predict responses in each cortical voxel. We test this via a lcave-twooout experiment. where we train a learner using 98 (Iy. ((z)).t)) pairs and then use this learner to predict which of the remaining two representation embeddings match the remaining two performance vectors. If the correlation between the predicted performance vector for representation A matches the ground truth performance vector for representation A better than the ground truth performance vector for representation B (and similarly B matches B better than it matches A), then we have succeeded in correctly discriminating performance vector from its corresponding representation embedding.
For each subject æ and pair of language representations (t,, ty), we trained a linear regression h( to map representation embedding r& to the corresponding encoding model performance built from t over all voxels.
( corr(g .k) (f4), B(,,,) P(x,eite)>
Fs defined as the 196-element vector that concatenates the row and column that correspond the representation t., leaving out the four elements corresponding to the held out pair: Ru. Ry, Rj, and
8

---

PMUE
CPEON Sage BPAT AAMEE Sepydl Regoda
Discriminabilik Score
Sepresenttiin
Figure 4: Representation Embeddings Reflect Brain Responses: (Left): Discriminability score matrix M, the average across each subject matrix M.,, which is computed described in Section 4.2.4. For each representation, we fit and tested encoding models that predict the fMRI response in each cortical voxel. yielding a pattern of prediction performance across the brain. For each pair, we then tested whether the brain patterns could be correctly matched to the representations on the basis of the representation embeddings shown in Figure 2. Highly discriminable pairs appear red, non- discriminable pairs white, and pairs that are less discriminable than expected by chance appear blue. Most pairs of representations yield brain patterns that are easy to distinguish using the embeddings, suggesting that these embeddings reflect the structure of representations in the human brain. However, some pairs are similar enough both embedding and brain that discrimination between them falls to chance level, such the word embeddings (green labels) and nearby layers of language models. (Upper Right): The percentage of pairwise matches for each representation where the match is correct more often than not (on 3 or more subjects). Almost all representations can be correctly matched with their corresponding performance vector p. with the interpretable representations being the most difficult to distinguish. (Lower Right): Mean voxelwise correlation for encoding models built using each representation of the natural language story stimuli. As seen in other literature 30,, the intermediate layers of Transformer- based language models work best encoding model representations.
Ryj. The model h(s.t..t)) was then used to predict the held out encoding model performance across all voxels, p(z, o, t4) and p(z, e, ty). For each subject, we then computed discriminability scores for all pairs of representations to generate matrix Mg. This quantifies how much better the predicted encoding model performance for t4 matched with the true encoding model performance for t; than the true encoding model performance for ty (and vice versa):
M =(corr(h(a.ttd)) P(a.odt)) + (corr(hesstat)()))) P(a.a)))))))))))) (coor(((s..t.t)) (r.), (())) + (corr((i,.t,d) P(z.))))))
Figure shows the discriminability scores for all representation pairs averaged across subjects, as well as the average encoding model performance for each individual representation. The vast majority of pairs have high discriminability scores (red), indicating that the representation embeddings broadly capture the structure of the brain's 's language representations.
However, few pairs have negative scores (blue), indicating below-chance discrimination perfor- mance. This could happen by chance, or it could happen if the task affinities are very different from the brain representations. We believe the negative scores for GloVe NWE are caused by the latter. This representation is highly dissimilar from the others, hence its far-removed location the MDS space. However, the temporal imprecision of the BOLD responses recorded by fMRI (on the order of I second, or 2-5 words) renders the GloVe NWE and normal Glo Ve embeddings nearly identical for predicting brain data. This disconnect is likely the cause of the negative scores.

---

We also computed how often the discriminability score was higher than 0 in the majority of subjects (at least 3 out of 5). We see (Figure 4 upper right) that encoding model performances are well predicted by the representation embeddings, with representation embeddings getting matched to their corresponding encoding model performances over 90% of the time for all representations, and 100% of the time for the majority of representations.
Future Work, Limitations, and Conclusions
By measuring transferability between different language representations, we were able to visualize the relationships between 100 representations extracted from common NLP tasks. This uncovered a www-dimensional structure that illustrates how different tasks are related to one another, and how representations evolve over depth in different language models. Finally, we show evidence that these representation embeddings capture the structure of language representations in the human brain.
There are further possible iterations of this method that may shed additional light the structure of language representations in the brain. Since this method relies on using the transfer properties between representations to generate embeddings, it is necessary that the chosen set of representations spans the rich space of language representations. If some part of the space of language representations has not been sampled, then the representation embeddings may fail to capture some aspects of the structure of that space. We tried to ameliorate this concern by sampling a large number of representations (100). However, it is likely that these chosen representations still do not capture the full diversity of the space. In future work, an even larger number of representations could be used to better map this space.
In a similar vein, we selected GloVe embeddings as the "universal" input representation for our encoder-decoder setup. Glo Ve embeddings are likely not "universal" in the same way that one-hot encoding of the story might be, but practical concerns made this approximation necessary. Use of other universal spaces could also be enlightening, such as directly using IMRI data map to a latent representation space-i.e. replace US) in Figure with IMRI BOLD responses B recorded from the same natural language experiment. Of course, the possibilities are not restricted to language representations, as previous work had focused on vision instead 39 44).
We also used linear as opposed nonlinear networks for encoder-decodens This limits the expressiveness of the transfer networks, and thus may under-estimate the relatedness between tasks. However, using only linear networks also increases interpretability by ensuring that only simply- related representations can transfer to one another, as also noted by others [2] 26]. Future work may explore the use of regularized nonlinear models for the encoder-decoder networks.
This method does require extracting representations from a neural network trained to accomplish some task. For many tasks used in linguistics, neuroscience, and cognitive psychology, there is no existing pre- traitreied network. However, training artificial neural networks on a task that used to measure cognition in humans and animals is an emerging method that can shed light on how
these tasks are accomplished [T 35| 42). and provide standardized benchmarks for comparing between different computational cognitive models. Furthermore, to understand what representations are needed to accomplish different tasks, and to visualize the relationships between those representations. For example, the debate whether lexico-semantic and syntactic representations are actually separable [14] centers on evidence from different tasks that claim to measure syntactic representations versus semantic ones. The claims about the tasks themselves are difficult to test, but methods like those presented in this work can directly quantify how much information is shared between representations needed accomplish those tasks, either in artificial neural networks or in neuroimaging data. Our proposed method may even provide new evidence in other domains where overlap has been observed between neural representations elicited by ostensibly different cognitive tasks 38].
use our proposed method
In sum, believe that our work can provide a template for investigating relationships between linguistic and cognitive representations in many different domains.
Acknowledgements
We would like to thank Amanda LeBel for collecting the fMRI data and reconstructing cortical surfaces, Lauren Wagner for annotating the experimental stimuli, and the anonymous reviewers for their insights and suggestions. Funding for this work was provided by the Burroughs Wellcome Fund
10

---

Career Award at the Scientific Interface (CASI), Intel Corporation, the Whitehall Foundation, and the Alfred Sloan Foundation. The authors declare no conflicts of interest.
References
[1] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54-59, 2019,
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 1600v:1610.0166 644, 2016.
[3] Edward Awh and John Jonides. Overlapping mechanisms of attention and spatial working memory. Trends in Cognitive Sciences, 5(3):119-126, Mar 2001. ISSN 1364-6613, doi:
(4.0066/113644-61330001113333331
[4] Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. Where is the semantic system? critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2677277%, 2009.
[5] Charlotte Caucheteux and Rémn-Rémi King. Language processing in brains and deep neural networks: computational convergence and limits. bioRxiv, 2020. doi: 10. 101002222000001 186288. URL htttps://wwe www.:biorxiv.oor o/t. 186288
[6] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transfommer-x:: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02866 2019.
[7] Wendy A de Heer, Alexander G Huth, Thomas L Griffiths, Jack L Gallant, and Frédéric E Theunissen. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27):65399665 6557, 2017.
[8] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391-40) 1990.
[9] Christophe Destrieux. Bruce Fischl, Anders Dale, and Eric Halgren. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage, 53(1):
1-15, 2010.
[10] Jacob Devlin, Ming-Woe Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
(11) Daniel B. Ehrlich, Jasmine T. Stone, David Brandfonbrener, Alexander Atanasov, and John D. Murray. Psychrnn: An accessible and flexible python package for training recurrent neu- ral network models cognitive tasks. eNeuro, 8(1), Jan 2021. ISSN 2373-2822. doi: URL UtLpttts::// wttphhttt///wwt eneuro. wwwo//ggoootooooooooo http://onntttt///////eee
10. N0.552MENEURR.0042720000
0427-20.202 2020
[12] Kawin Ethayarajh. How contextual contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint 1rXiv: 2019.
[13] Evelina Fedorenko, John Duncan, and Nancy Kanwisher. Language-selective and domain- general regions lie side by side within broca's area. Current Biology, 22(21):2059-200 2012.
[14] Evelina Fedorenko, Idan Asher Blank, Matthew Siegelman, and Zachary Mineroff. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition, 203, Oct 2020. ISSN 0010-0277. doi: 10. 2a..016f.cogmition2200....00
11

---

[15] Bruce Fischl. Freesurfer. Neuroimage, 62,)
[16] Gregory Hickok and David Poeppel. The cortical organization of speech processing. Nature reviews neuroscience, 8(5):393 402, 2007.
[17] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant, Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453-44) 2016.
[18] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. httphtttpp/ooceeeeeeeee wewpeeeddinnnnnnnnninneeeeeeeeee www.neuuiip... httpa//g...riiissc////////////// cctppaaa/eep//////i/ee///
.d
[19] Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexander Huth. Interpretable multi-timescale models for predicting fmri responses to continuous natural speech. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 13738 13749. Cur- ran Associates, Inc., 2020. URL/attts://poooeedini neurips httppps.ccpppppp/pp http/ppppr2220/ff1/p/
[20] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning, pages 3519-3529, PMLR, 2019.
[21] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909. 11942, 2019.
[22] Yulia Lerner, Christopher Honey, Lauren Silbert, and Uri Hasson. Topographic mapping of hierarchy of temporal receptive windows using narrated story. Journal of Neuroscience, 31 (8):2906-22) 2915, 2011.
[23] Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21411):164111646, 2011.
[24] Shinji Nishimoto, Alexander G Huth, Natalia Y Bilenko, and Jack L Gallant. Eye movement- invariant representations in the human visual system. Journal of vision, 17(1):11-1,, 2017.
[25] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
(26) Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-thhoredi probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, 2020.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
[28] R.W. Saaty. The analytic hicrarchy process- what it and how it is used. Mathe- matical Modelling, 93)::111-776 1987. ISSN 0270-0255. doi: https://oi.org//0.000
0270-0255187)99473--9 URL https: //w::://www.w/ www.:s.ceecnediiec...... com.s///reetttcccciiiiiiccccccee wttp.///ccciinceearrii.... p11/0270025587904778
[29] Helmut Schmid. Part-of-speech tagging with neural networks. arXiv preprint cmp-1g/9410018, 1994.
[30] Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Inte- grative reversi-n converges model for predictive processing. bioRxiv, 2020. doi: 10. 10.1101//020.0.222 174482. URL. https: www biorxiv. http://oog/////eeeeeeeeeeeeeeeee arg//onteen//earyy/2000 10/09/ 20000 06. 174482
12

---

[31] Youwei Song, Jiahai Wang, Zhiwei Liang, Zhiyue Liu, and Tao Jiang. Utilizing bert intermediate layers for aspect based sentiment analysis and natural language inference. arXiv preprint arXiv:2002.048155 2020.
[32] Jörg Tiedemann and Santhosh Thottingal. OPUS-MT the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal, 2020.
- open translation services for
[33] Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processingg (in the brain). In H. Wallach, H. Larochelle, A. Beygelzimer, d' AlcchéBBue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. https:: http://////oeeeeeeeeeeeeee .
[34] Warren Torgerson. Multidimensional scaling: theory and method. Psychometrika, 17(4): 401-419, 1952.
[35] Ben Tsuda, Kay M. Tye, Hava T. Siegelmann, and Terrence J. Sejnowski. A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. Proceedings of the National Academy of Sciences, 1 117447):98722-29882 Nov 2020. ISSN
0027-8424, 1091-6490. doi: 00.1073/pnas.200959111
[36]
https://themen org
The moth radio hour, 2020.
[37] Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Matkarella-Micke, Subhransu Maji. and Mohit Iyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882-7926, Online, November 2020. Association for Computational Linguistics. doi: 10.
URL https://wwe
aclweb. org/ anthology/2020. emn connp-mminn...m.mmmmmmmmmmmmmmmm
[38] Vincent Walsh. theory of magnitude: common cortical metrics of time, space and quantity. Trends in Cognitive Sciences, 7(11):483-488 Nov 2003, ISSN 1364-6613, doi: 10.1016/jj.ics.. 2003.09.002.
[39] Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d''lché-Buuc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
neurips neurips pdf
[40] Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved different story reading subprocesses. PloS one, 9(1 I):el 2014.
[41] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers:
ofateoof-hhe-ar natural language processing. ArXiv, abs/1910.03771. 2019.
[42] Guangyu Robert Yang, Madhura R. Joglekar, H. Francis Song. William T. Newsome, and Xiao- Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience, 22(2), Feb 2019. ISSN 1546-1726. doi: 10.
[43] An Gie Yong, Sean Pearce, al. A beginner's guide factor analysis: Focusing on exploratory factor analysis. Tutorials in quantitative methods for psychology. 9(2):79-94, 2013.
[44] Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
13


Reviews:
[{"review_id": "yPaONfWm5hL", "reviewer": "Reviewer_AEZD", "summary": "* The authors propose \"representation embeddings\" (vectors describing transfer performance from task i to task j) as a low-dimensional space describing the content of language representations.\n* They analyze the derived representation embedding space and claim that the main principal component tracks \"depth\" (line 192) of linguistic processing.\n* They conduct a brain encoding task and evaluate 100 models on this task. They claim that a model's brain encoding performance tracks with its behavior in this low-dimensional representation embedding space, and that this low-dimensional behavior also maps intuitively onto the spatial representation of language in the brain.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is exceptionally well motivated among brain encoding papers, and clearly written. However, I think the results and conclusions fall short of this motivation. I think the results are marginally worth publishing as-is, but the paper could be strengthened by addressing some of the conceptual issues below.  \n\n  \n### Quality/significance\n  1. I strongly support the authors' mission to move past coarse, qualitative labels such as \"syntax,\" \"semantics,\" and \"high/low-level\" in the quest to understand both neural network and brain representations of language. But unfortunately the conclusions of the paper still seem to entirely depend on these pretheoretical notions. The MDS analysis is mainly framed as interesting because its first principal component seems to recapitulate the syntax--semantics processing trajectory (lines 183--187). The brain encoding analysis is also framed as a success because it matches up with existing notions of the spatial distribution of \"higher-order\" processing in the brain (line 266).As such, I think the original motivation of the authors is not fully accomplished in this paper. Ideally we would be able to use novel quantitative tools to go beyond our current conceptual framework (e.g. proving their worth in exploratory data analyses, or making predictions at a granularity / in a domain not afforded by the original qualitative distinctions).Suggestion for improvement: Explain concretely in your 'future work' section how this paper (or the next paper) can go beyond the qualitative distinctions of \"syntax,\" \"high/low-level,\" etc. \n  2. The neuroscientific contribution is not clear to me. The authors claim to test whether the \"MDS dimension could capture patterns of hierarchical processing observed in cortex,\" and claim that the resulting voxel map of Figure 3 answers in the affirmative. But without a more specific (quantitative / statistical?) statement of the claim and without any reference / comparison to alternative hypotheses, it's not clear what success or failure should look like for this evaluation.Suggestion for improvement: Evaluate against a quantitative baseline performance measure, or design (post-hoc?) statistical tests to make concrete some of your localization claims.\n  3. I admire the creative use of higher-order \"representation embeddings\" here, and the mapping of its principal components back onto higher-order features of brain encoding models. But I have trouble seeing a path forward for the cognitive neuroscientific project here. This analysis setup seems to only be capable of mapping between-model representational distinctions (e.g. shallow vs deep linguistic processing) brain areas. The high-level analysis erases by design any between-stimulus distinctions; as such, it blocks us from exploring more advanced questions about fine-grained distinctions in linguistic representational content (as random examples -- how/when/where is implicature computed, how/when/where are scope ambiguities resolved, etc. etc.). Because the authors seem to be suggesting this sort of analysis as the beginning of a broader research program, I would appreciate if they could address the limits of the questions they can answer, in terms of representational content.Suggestion for improvement: Add discussion to conclusion.\n\n### Clarity\n  1. I had a very difficult time understanding both the motivation for and the mechanics of the discriminability analysis of section 4.2.4\\. I don't understand what this analysis contributes beyond the more basic encoding results in Fig. 4 lower right. Why is this measure computed on pairs of representations? And why is a high discriminability evidence of \"broadly captur\\[ing\\] the structure of the brain's language representations,\" over and above the basic correlation values?Suggestion for improvement: I would definitely appreciate more motivation of the particular method, which is currently very difficult to understand. (It's a second-order analysis of brain encoding models trained on second-order language representations!)\n  2. Minor comments/questions\n    * Line 261: What is the motivation for z-scoring these values?\n    * Section 4.2.4, M equation: Why are you computing correlations here? My understanding is that both $\\hat{h}$ and $\\rho$ are nonrandom scalar values.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above for scientific limitations. No issues re: societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "O0LaKR1tWGE", "reviewer": "Reviewer_KDff", "summary": "Using a slightly modified (but relatively straightforward version) of an analysis that’s recently been called ‘neural taskonomy’, this work attempts to use the relationships (read: transfer learning affinity) among different natural language representations to assess the organization of linguistic representation in human fMRI data via regularized linear encoding models.  The authors first contribute a modified version of a previously used encoder-decoder framework to create a 'task' (linguistic representation) affinity matrix that demonstrates conspicuous, interpretable structure when reduced via MDS to fewer projections. They then use the structure discernible in this matrix to ascertain whether similar structure (specifically, a hierarchy of representation from word embeddings to later, deep transformer layers) is evident in the brain. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ‘taskonomy’ portion of this paper (in that it closely mirrors the work of Zamir & Colleagues) is mostly clear and well-constructed; the selection of models seems comprehensive enough (spanning relatively simple word embeddings to more complex transformers), and the range of linguistic tasks covered seems (at first glance) sufficient to recapitulate the diversity of semantic and syntactical structures that should presumably define the neural code. Some minor points of clarification I think could benefit this section are:\n- Clarifying what exactly is meant by the ‘mean squared error’ on some % of the data in the generation of the pairwise tournament matrix (135-139). How is the data being divided? What exactly is the data in this case? A particular set of sentences in the corpus being tested? This section – though perhaps routine in principle – left me utterly complexed.\n-Reformulating the sentence starting at line 161 (“We also included the GloVe 162 embedding for the next word in the sequence, which constitutes a low-dimensional representation the ideal output of a language model”) to both remove the typo (assuming it’s missing a colon) and to clarify why a low-dimensional representation is the ideal output of a language model. This is not immediately clear to me a priori, and I might even be inclined to say the opposite, depending on the functional context in which the representation is leveraged.\n-Is the vertical (top to bottom) dimension in the MDS interpretable? The EFA suggests the second most significant factor still accounts for 10% of the variance. Does this 10% come with interpretability? If not, is it really the case that the low-dimensional structure among the natural language representations is interpretable in only one direction?\n-What is so different about the GLOVE Next Word embeddings? Why their consignment to the periphery of the MDS?\n\nAll in all, I find this section particularly compelling, and am intrigued by the structure the authors seemed to have uncovered – though an attempt at interpreting other dimensions after the 1st could make it just a bit more satisfying.\n\nThe ‘neural’ portion of this paper is the less compelling of the two – not necessarily due to methodological flaws, but to a lingering sense that some of the author’s central claims (including, most importantly, the recapitulation of the taskonomic representational hierarchy across the brain) are not so neatly supported by the data. Take for example the BrainView in Figure 3. The authors make a significant point of underscoring a hierarchy of representation (from word embeddings to late transformer layers), but most of the areas activated by the ‘later’ (presumably more complex) representations are early perceptual areas (including most conspicuously, early visual cortex). Anterior inferotemporal cortex (the main site of object recognition, and presumably the location where vision interfaces with language) is dominated by ‘earlier’ (presumably less complex) representations. I can’t quite make sense of this in a narrative about representational hierarchy – but this brings me to a second point: the authors do not make clear what we should be looking for. Are there known representational hierarchies in the neuroimaging of language? If so, how are they characterized and what are their stereotypical motifs? Obviously, the specific loci of syntactic and semantic structures in the brain remain a matter of significant (even hot) debate – but some reference point here (from Caramazza, Federenko or other similar researchers) could help to situate and better direct the reader’s attention to key data points of high interpretive value. \n\nSimilarly, the lack of an overall gestalt in Figure 4 (at least a gestalt comparable to that provided by the MDS plot in Figure 2) gives the impression that the authors relied a bit too heavily on a data-driven approach without a clear sense of the theoretical insights to be gained from their otherwise powerful methodologies. What does it mean that the GPT models were overall most predictive of neural activity? Is there discernible structure in the MDS computed on the discriminability score matrix (perhaps structure comparable to that found for transfer affinities)? For that matter, what do all of these answers look like in parts of the brain localized or functionally segmented with a language-specific task? Without answers to these questions, I find it difficult to say what I’ve learned from the ‘neural’ half of this ‘neural taskonomy’ analysis. \n\nI think this paper has significant potential, and would be happy to add to my score if the authors can more clearly address the following two points in their rebuttals:\n- Is there a distinct linguistic hierarchy in (either the whole or a functionally localized linguistic portion of) the brain? If yes, how is it defined, where exactly is it located and does the hierarchy unveiled by your task affinities match this hierarchy?\n- What is the overall takeaway from the rankings of the 100 representations in terms of their power as predictors in neural encoding models? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes -- though perhaps some thoughts on the limitations of purely data-driven approaches to computational neuroscience would have been apropos.", "review_text": "", "extra_scores": {"ethical_concerns": "The authors adequately addressed ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ATzrSIyxtvx", "reviewer": "Reviewer_u2XG", "summary": "This paper is about two things: understanding learned NLP representations by finding and understanding similarities between *different* representations; and comparing these representations to fMRI data. The paper takes a new approach to both problems, and finds a dimension in the space of language representations that seems to correspond to something like \"semantic depth of analysis.\" Furthermore, the comparison with fMRI data shows that many representations are correlated with brain activations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very interesting methodologically, in the sense that it brings novel methods to an important problem, and also in the sense that these methods are highly worthy of further examination. I don't feel that I learned much about NLP representations or about brain representations, but I do feel like the approach taken here is going to be an important part of answering these questions in the future.\n\nThe paper is very clear, but there were still some parts that I had difficulty with. First of all, the methods in 3.3. It seems complicated: if the goal is to get a vector, for each type of representation, that encodes in some sense how similar another representation is, then why go to the trouble of the \"tournament\"? Why not just, for each element i, directly use the MSE for decoding the target representation using the representation i? And, for that matter, even in the tournament version, why not take into account the actual values of the MSEs rather than just the percentage for which the one was higher than the other? Given that there is a simpler idea at hand, I would guess that the tournament approach must be necessary. Otherwise the paper would have presumably taken a simpler approach. But I do not understand why. This leaves open the possibility that this approach was chosen empirically, which wouldn't be a good thing here. More explanation of these methods and whether and why they *must* be the way they are, is necessary.\n\nMore information would be welcome about what the \"temporal transformation\" is in 4.2.2.\n\nIn 4.2.3, the methodology is sufficiently complicated that I have a hard time following what exactly \\rho is. More importantly, I have a hard time understanding why the \\rho values should be sufficiently comparable to the representation embeddings that it would make any sense to project them onto the first basis vector of the MDS. I have a very strong feeling that it makes sense,but this part of the paper just goes too fast for me to understand exactly with these p vectors are.\n\nSimilarly, I have a hard time getting the intuitions behind the method in 4.2.4, and, as such, I don't feel completely at ease interpreting Figure 4. For one thing, what do the negative scores in the upper left subfigure mean?\n\nAn interesting but controversial property of this approach is that it tries to map the linguistic embeddings down to a fixed dimension using a linear mapping. This is interesting, but surely the only risk is not *underestimating* the relatedness between tasks. Imagine in the worst case that all information is lost in the mapping, and we always get the zero vector, for two different representations. Surely in that case we should have great performance decoding and thereby *overestimate* the relatedness. \n\nI'll return to my remark that I'm not sure I learn a lot empirically - although, I stress, this paper does have a lot to recommend it, particularly methodologically. I don't really see much in the brain map in Figure 3. The text tells me there's something - not necessarily much - and I guess I trust that, but it does not seem like there's much there. And I have a hard time getting *anything* useful out of Figure 4, aside from the fact that these representations predict voxels. If there is more, and more interpretable, content to be had in these results, I'd encourage the authors to draw it out more. Similarly, the similarity matrix in Figure 2 isn't all that informative, although the MDS is certainly intriguing. However, it leaves a lot of open questions. Is this first component really \"semantic depth\"? What further tests could we do to probe whether that's the case? How could we operationalize that notion? Second, what is this second dimension doing? And, while the paper suggests that the later layers of some models come close to the next word embedding, I don't see them (visually) coming very close, and a glimpse at the similarity matrix in Figure 2 confirms that for me. The next word embedding is really far from everything. So... doesn't that mean these language models are bad?\n\nIn short, I think there is a lot of promising exploration made possible by this paper. I'm not sure there is enough exploration and discussion, as per my comments. And I find the methodology either too complicated, too sparsely motivated, or just hard to follow because it's too dense. \n\nComments in author review period\n===========================\n\nMy overall assessment of this paper is exactly the same, except that I realize that the methods (based on Zamir) are slightly less original than I had thought. I am lowering my score to a 7 to reflect this. It is still a very interesting if preliminary exploration.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, though see above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "3HTq4ADNZiP", "reviewer": "Reviewer_aJCH", "summary": "The submission proposes to study similarities between language representations in different language models. First, a similarity measure between language representations is derived. Then, the matrix of similarities is reduced to two dimensions using MDS. The authors then discuss qualitatively this low-dimensional space of language representations. Then, linear encoding models of fMRI recordings are used to project the first MDS dimension on the cortical surface of a subject listening to narratives. Finally, the matrix of similarities is used to predict the performances of linear encoding models.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality:* Comparing representations from intermediate layers of deep neural networks is an interesting topic and has received a lot of interest in the recent years, as adequately discussed in the introduction. This submission proposes a new method to compare representations, using a procedure based on bottle-necked encoder-decoder across representations, creating tournament matrices from pairwise comparisons of representations, and stacking the tournament matrices first eigenvectors to create a similarity matrix. This procedure seems partially novel and partially inspired from Zamir et al [38], although it is not clear how this work exactly relates to previous contributions. In particular, there is no quantitative nor qualitative comparison with what would be obtained with CCA or CKA. \n\n*Quality:* The submission is technically sound, although the motivation of each step of the procedure could be better explained. The results look reasonable and robust, and adequate care about generalization and overfitting is mentioned. The authors note the limitation of centering their similarity measure around the GloVe word embedding. The shared code and data is appreciated, although not all code files reach a reasonable level of documentation. Importantly however, the authors do not compare their procedure to other similarity of representations methods .\n\n*Clarity:* The paper is clearly written and easy to follow, except a few convoluted sentences (e.g. L36, L74). However, the paper could better inform the reader about what parts of the method to build the similarity matrix is novel. The discriminability scores in section 4.2.4 are not enough motivated and explained. In particular, it is not clear why \"high discriminability scores (red) [indicates] that the representation embeddings broadly capture the structure of the brain’s language representations\". \n\n*Significance:* It is hard to judge the significance of the method section (section 3), since no comparison with previous work is proposed. However, using MDS on the similarity matrix and the corresponding discussion around Figure 2 is interesting. The projection over the cortical surface is not surprising, as the first MDS dimension is mainly a measure of the DNN layer, and similar gradients have been described over encoding models built from DNN layers [e.g. Kell et al, Neuron 2018].\n\n*remark:* NLP is not defined.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found that the method presented in this submission is useful and casts our understanding of language processing in the brain in a new light.

I encourage the authors to update their manuscript to include the results from the discussion with the reviewers, as these were decisive.

Decision: Accept (Poster)