# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Composing Word Vectors
#### 2.1.5 Clustering  
As shown by [71], averaging word vectors leads to a loss of information. It is in particular significant for longer text sequences, such as documents. Therefore, the idea was developed to aggregate vectors in a more smooth and information-preserving manner. The authors of [72] clustered the word vectors into \(k\) groups using \(k\)-means. Then, for a document, each cluster vector is obtained as a sum of its constituent vectors. The final representation is then the concatenation of the cluster vectors and the inverse cluster frequency (icf) values, which are calculated using the idf values of the words present in the document. Other authors of [73] proposed a similar method, but viewed it as one that reduces the dimensionality from words to concepts. Concepts (as clusters) are created by clustering word vectors generated from Word2 Vec, and frequencies of these concept clusters are used to represent document vectors. Finally, similarly to tf-idf applied to bag-of-words, a weighting scheme, concept frequency-inverse document frequency (cf-idf), is applied to acquired bag-of-concepts. The authors of [74] proposed the concepts of word containers and document containers to explain how such procedures bring benefits. Similarly to other works, they perform clustering of words into distinct clusters. Vectors belonging to the same cluster are averaged, while resulting representations among different clusters are concatenated. Using standard contextualized pre-trained transformer models with frozen weights, the authors of [75], instead of a simple mean pooling, proposed training a categorical variational autoencoder and reported the performance improvement on some STS tasks. This way, similar to clustering, the lower intermediate compressed representation would carry the essential representation of the sequence. Overall, using clustering of independent word vectors allows the document representation to be expressed in a smaller space of concepts, whose concepts are otherwise erased due to destructive interference during the global averaging. The authors of [76] improved the aforementioned method of [72] by using soft clustering and allowing a single word vector to participate in multiple clusters. More specifically, each word is represented as a \(K\times d\) dimensional embedding, where each \(k^{\text{th}}\) row corresponds to the original word vector weighted by its probability distribution in the \(k^{\text{th}}\) cluster. The embedding of this word is weighted with the inverse document frequency of the word and summed with the embeddings of other words to form a document vector. As many values in such a vector were observed to be close to zero, sparseness is induced with a given minimal value threshold. The authors of [76] named their method Sparse Composite Document Vector (SCDV). The success of SCDV was supplemented by numerous other contributions. The authors of [77], instead of the Gaussian Mixture Models (GMMs), used K-SVD (an algorithm for designing overcomplete dictionaries for sparse representation) [78] for the topic modeling. Furthermore, they used a newer word vector algorithm Doc2 VecC and a modern SIF weighting and removal of the top principal component of [60]. Other authors of [79], with their SCDV-MS method, additionally performed word sense disambiguation to dissect polysemous words into distinct vectors. In this way, the quality of the clusters improved. To induce sparsity, the authors also applied hard thresholding in an earlier stage in word cluster assignments and trained word vectors with Doc2 vecC. Next, with the emergence of transformer models, the authors of [80] utilized contextualized representations in their SCDV + BERT (ctxd) method. First, the corpus is contextualized following the technique of [81]. That is, each corpus word is clustered among its individual recurrence contexts into distinct clusters corresponding to the different meanings of the same word.