# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Reshaping Representation Spaces
#### 2.2.1 Isotropy  
Isotropy is defined as uniformity in all orientations. In this work, we talk about embedding spaces of words and sentences. According to the authors of [117], \"distribution is isotropic when the variance is uniformly distributed across all dimensions\". The anisotropic properties of independent word embeddings were first observed by the authors of [118]. They analyzed common embedding models and found that, in all cases, word vectors share a nonzero common vector; therefore, they are not zero-centered. The authors also found that the variance ratios of the first few components derived by PCA decay nearly exponentially. Furthermore, the variances explained by the leading components \"encode the frequency of the word to a significant degree\". Therefore, they proposed eliminating the common mean vector and then removing the top principal components, computed on representations from the entire vocabulary. A similar procedure was also shown in an earlier work [60] to be a very good baseline. Here, words were combined by a frequency-weighted average, and then just the first principal component (yet dataset-specific, as computed on the entire dataset) was removed. As shown by the All-But-The-Top (ABTT) method of [118], such a simple procedure, eliminating the common mean vector and a few top dominating directions from the word vectors, greatly improves both the isotropy and the downstream task performance (for sentence tasks, a simple average of preprocessed word vectors is used). As found by [119], it is already known that the isotropy of the target embedding is critical for the alignment of embeddings, which is important for areas such as domain adaptation, word embedding evaluation, and machine translation. However, the work [118] became the main stimulus for the research direction that attempts to improve the isotropy of word embeddings. Word representation anisotropy is also observed in the latest contextualized models. The authors of [120] first observed this effect on the transformer machine translation model [2], while the author of [98] found that this is the case for all ELMo [106], BERT [5], and GPT-2 [121] layers. All representations for randomly sampled words from these contextualized anisotropic spaces were observed to be highly similar, far from zero average cosine similarity. Furthermore, the embeddings of any two words are positively correlated. Finally, it was observed that all word vectors in the representation space tend to occupy a narrow cone. The research community offered some explanations for why anisotropy may occur. We will cover them in more detail below. The authors of [122] were among the first to observe that word embeddings encode a surprising degree of information about word frequencies. In [60], it was shown that simply representing sentences as a weighted average of words by their frequencies is a very competitive method. In particular, the authors computed the weighted average of the word vectors in the sentence (the weight of a word \(w\) is _a/(a + p(w))_, with \(a\) being a parameter and _p(w)_ the frequency) and then removed the projections of the average vectors on their first singular vector (\"common component removal\"). Later, the authors of [118] suggested removing more than one component. They found that the top PCA directions encode word frequency information; therefore, the method was proposed to null these directions. As this bias is very strong in representation space, it is inevitably related to the anisotropy phenomena. Similar encoding of frequency information is also evident in contextualized embeddings of modern transformer models. The authors of [123] investigated BERT embeddings and found that high-frequency words are all close to the origin and densely concentrated,while low-frequency words are far away and scattered sparsely. The work in [124] highlighted that anisotropy is the most pronounced in rare words.