Low-dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

1 Introduction . There are a multitude of common techniques for analytically representing the information contained in natural language . At the word level , language is often represented by word embeddings , which capture some aspects of word meaning using word co-occurrence statistics [ 8 , 25 ] . Language representations that highlight specific linguistic properties , such as parts-of-speech [ 29 ] or sentence chunks [ 1 ] , or that utilize well-known NLP models such as the intermediate layers of pretrained language models [ 6 , 10 , 21 , 27 ] , are also frequently studied [ 12 , 31 ] . In fields such as linguistics , natural language processing , and cognitive neuroscience , qualitative adjectives are often used to describe these language representations – e.g . “ low-level ” or “ high-level ” and “ syntactic ” or “ semantic ” . The use of these words belies an unstated hypothesis about the nature of the space of language representations – namely that this space is fundamentally low-dimensional , and therefore that the information from the representations in this space can be efficiently described using a few categorical descriptors . In this work , we attempt to directly map the low-dimensional space of language representations by generating “ representation embeddings ” using a method inspired by the work of Zamir et al . [ 44 ] . This method uses the transfer properties between representations to map their relationships . The work described here has two main contributions . First , we used the representation embeddings to demonstrate the existence of low-dimensional structure within the space of language representations . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We then used this structure to explore the relationships between – and gain deeper insight about – frequently used language representations . How do the intermediate layers of prominent language models relate to one another ? How do the abstractions used by these layers evolve from low-level word embeddings of a context to a representation of the predicted next word for that context ? Do different language models follow similar representation patterns ? What are the differences between how unidirectional and bidirectional language models represent information ? These are examples of the types of questions we can explore utilizing our representation embedding space . Second , we showed that this low-dimensional structure is reflected in brain responses predicted by these representation embeddings . In particular , we show that mapping the principal dimension of the representation embeddings onto the brain recovers , broadly , known language processing hierarchies . We also show that the representation embeddings can be used to predict which representations map well to each area in the brain . 2 Related Work . Our work closely follows the methods developed by Zamir et al . [ 44 ] . Those authors generated a taxonomy of vision tasks and analyzed their relationships in that space . This was achieved by applying transfer learning across vision tasks , and showed that this can reduce data labeling requirements . We used some of their methods , such as transfer modeling and normalization using an analytic hierarchy process ( AHP ) [ 28 ] , for our analysis on language tasks . Following the same lines but for NLP , the work from Vu et al . [ 37 ] explores transferability across NLP tasks . Those authors focus on the application of transfer learning to show that the amount of labeled data , source and target tasks , and domain similarity are all relevant factors for the quality of transferability . Kornblith et al . [ 20 ] investigated and compared techniques for measuring the similarity of neural network representations , such as canonical correlation analysis ( CCA ) and centered kernel alignment ( CKA ) . Our analysis of the relationship of between the language representation space and the brain leans heavily on the concept of “ encoding models ” . Recently , encoding models have been widely applied as predictive models of brain responses to natural language [ 17 , 40 ] . Jain and Huth [ 18 ] applied encoding models to representations of an LSTM language model that incorporates context into neural representations . These representations are explored for different timescales of language processing in Jain et al . [ 19 ] . Toneva and Wehbe [ 33 ] explored the encoding model performance of different language model layers to improve neural network performance . Caucheteux and King [ 5 ] observed differences in the convergence of hidden state representations from ANNs trained on various visual and language processing tasks to brain-like representations . Schrimpf et al . [ 30 ] systematically examined the performance of encoding models across a variety of representations and neural response datasets and observed consistent high performance of Transformer-based model hidden states at predicting neural responses to language . Wang et al . [ 39 ] constructed encoding models on a set of 21 vision tasks from Zamir et al . [ 44 ] with the objective of localizing tasks to specific brain regions . Those authors built a task graph of brain regions showing similar representation for highly transferable tasks . 3 Methods Overview . Our method aims to use the transfer properties between different language representations to generate quantitative descriptions of their relationships . These descriptions can then be used as an embedding space without labelling any individual representation with predefined notions of what that representation is or how it relates to other representations . This is analogous to how word embeddings use co-occurrence statistics to generate quantitative descriptions of each word without labelling any individual word with our predefined notions of what that word means . By using transfer properties to generate the embedding for each representation , we implicitly assume that language representations with similar transfer properties are themselves similar . We define a representation t as a function over a language input that extracts a vector of meaningful features for each word . Some representations can be computed independently for each word , such as word embeddings [ 1 , 10 , 25 ] , while others depend on both a word and its context , such as part of speech labels [ 1 ] , or the hidden state in some intermediate layer of a language model [ 5 , 6 , 27 , 30 ] . We define a stimulus s as a string of words drawn from some natural language source , and S the set of all stimuli from that source . To generate a representation embedding over a set of representations T that are derivable from the same stimuli S we will construct time-dependent mappings between representations . For a representation t ∈ T , and for all times j in our stimulus s ∈ S , the vector t ( sj ) is defined . Next , for each pair of representations t1 , t2 ∈ T we will attempt to map t1 ( sj ) to t2 ( sj ) . This provides a transfer mapping ft1→t2 ( · ) that can be learned for each pair of representations , ft1→t2 ( t1 ( sj ) ) ≈ t2 ( sj ) ∀j . However , learning these transfer mappings directly would make comparison across representations difficult , as many of the representations have different dimensionalities . Instead , we adapted the encoder-decoder framework proposed in Zamir et al . [ 44 ] , which ensures that every transfer mapping has the same input dimensionality . This method proceeds in three parts ( Figure 1 ) : First , we train a linear encoder for each representation that compresses the information in the input down into a latent space . The latent space should only contain information necessary to predict the given representation ’ s features from our input . This is accomplished by training an encoder-decoder from the input to the given representation via a latent space , then discarding the decoder . Second , we train a new decoder for each pair of representations that uses the previously generated latent space for one representation to predict the other representation . Third , we evaluate the performance of each decoder targeting a given representation relative to every other decoder targeting that same representation . This provides a measure of how much of the information in one representation is present in every other representation . Just as words that have similar co-occurrence statistics—when measured against a large enough group of other words—are thought to have similar meanings , we argue that pairs of representations that have similar transfer properties—when measured against a large enough group of other representations—have similar information content . 3.1 Generating Representation Encoders . We define1 U ( S ) as the universal input feature space for our stimuli S . In practice , this space should represent the input with high fidelity , so could be selected as a one-hot encoding of each input token , or word embeddings for each input token . We use GloVe word embeddings for U ( S ) . Now let T be a set of representations over S. For each representation t ∈ T , we generate an encoder Et ( · ) such that the encoder extracts only information in U ( S ) that is needed to predict t ( S ) . We do this by using a bottlenecked linear neural network that maps every u ∈ U ( S ) to an intermediate low-dimensional latent space Lt = Et ( U ( S ) ) and then maps it to the given representation space , t ( s ) ≈ f ( Et ( u ) ) ∀s ∈ S ∧ u = U ( s ) , where f ( · ) is mapping from Lt to t ( S ) . We used a small latent space of 20 dimensions to encourage the encoder to extract only the information in U ( S ) that is relevant to compute t ( S ) . Experimentation showed that variations in the latent space size do not meaningfully change encoder performance . 1We assume that elements j in a string sj are properly treated , such that we can exclude the positional indexes for all s. Thus , we define U ( S ) = { U ( s ) |s ∈ S } as the result of applying a function U ( · ) to all elements s ∈ S . Once we have learned these mappings , we assign Et to be our representation encoder for each representation t ∈ T . Regardless of the dimensionality of the representation , each latent space has a fixed dimensionality , which enables a fair comparison between representations . 3.2 Generating Representation Decoders . The encoders for each representation generate a latent space Lt that extracts the information in U ( S ) relevant to computing t ( S ) , while compressing away irrelevant information . For every pair of representations ( t1 , t2 ) ∈ T , we next generate a decoder Dt1→t2 such that Dt1→t2 ( Lt1 ) = Dt1→t2 ( Et1 ( U ( S ) ) ) approximates t2 ( S ) . This yields a total of n2 decoders , where n = |T | is the total number of representations . All networks were trained with batches of size 1024 and standard stochastic gradient descent with a learning rate of 10−4 for the initial encoders and 2× 10−5 for the decoders . Hyperparameters were chosen via coordinate descent . 3.3 Generating the Representation Embedding Matrix . Finally , we compare the performance of the n decoders for each representation in order to generate the representation embedding space . To ensure that comparisons are made on equal footing , we only compare decoders with the same output representation ( e.g . Dt1→t2 is compared with Dt3→t2 , Dt4→t2 , etc. ) . This is critical , because evaluating representations with varying dimensionalities would unfairly bias the results . This and related challenges were explored in detail in Zamir et al . [ 44 ] . We used the decoders to generate a pairwise tournament matrix Wt for each representation t by “ fighting ” all pairs of decoders that output to representation t using a held-out test set Stest of sentences . Element ( i , j ) in Wt contains the ratio of samples in the test set for which Dti→t has lower MSE than Dtj→t , i.e. , Wt ( i , j ) = Es∈Stest [ Dti→t ( s ) < Dtj→t ( s ) ] Es∈Stest [ Dti→t ( s ) > Dtj→t ( s ) ] . For example , if the decoder DA→C has lower mean squared error than decoder DB→C for 75 % of the data in Stest , we assign the ratio of 0.75/0.25 = 3 to entry ( A , B ) in the tournament matrix WC for representation C. We then use this set of pairwise comparisons to approximate a total order over the quality of all encoders for each decoded representation . This was done using the Analytic Hierarchy Process ( AHP ) [ 28 ] , a technique commonly used in operations research to convert many pairwise comparisons into an estimated total order . AHP establishes that the elements of the principal eigenvector of Wt constitutes a good candidate for a total order over the encoders for t. This eigenvector is proportional to the time that an infinite length random walk on the weighted bidirected graph induced by Wt will spend at any given representation . Thus , if the encoder for one representation is better than the others , the weight on that representation will be higher as compared to the weights of other encoders . This eigenvector is then normalized to sum to 1 . This procedure yields a length n vector for each of the n representations . We set a value of 0.1 into each vector at the position corresponding to the target representation , and then stack the resulting length n vectors together into the representation embedding matrix , R .

Summary: This paper is about two things: understanding learned NLP representations by finding and understanding similarities between *different* representations; and comparing these representations to fMRI data. The paper takes a new approach to both problems, and finds a dimension in the space of language representations that seems to correspond to something like "semantic depth of analysis." Furthermore, the comparison with fMRI data shows that many representations are correlated with brain activations.