# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Composing Word Vectors
#### 2.1.3 Averaging  
As neural vectors became more effective than traditional approaches [28], it turned out that it was common to derive a vector for a sequence of words simply by summing or averaging individual vectors. In 2009, the authors of [27] reported that for composinga phrase representation, \"averaging is the most common form of vector combination\". Despite its simplicity, the basic rule of averaging word vectors was shown to work very well [36; 37; 38]. Later work, such as [39; 40; 41], even showed that simple pooling methods, such as basic vector addition or averaging, match or outperform much more sophisticated methods for encoding the meaning of a text sequence. The authors of [42] reported that in out-of-domain scenarios, simple architectures such as word averaging outperform complex Long Short-Term Memory (LSTM) models. In addition, they are even competitive with systems tuned for particular tasks. Later work by [43] evaluated various compositional models and found that word vector averaging performed reasonably well in most supervised benchmarks. According to the authors of [9], even for the currently trending BERT transformer model, to map the sentence to a single vector, the most popular approach is to average the word embeddings of the BERT output layer. As averaging is well-suited to acquiring sentence optimization, some methods take averaging into their structure to further optimize it. One such example is the deep average network (DAN) [44]. It takes the input as an averaged text sequence and passes it to one or more feedforward layers to finally perform linear classification. Another, C-PHRASE model of [45], is trained to predict the contexts of phrases from the additive combination of their elements. Such a design results in a useful property of C-PHRASE, that summing word vectors yields sequence representation. The authors of [46] found that one should use an average of all hidden states of LSMT, rather than using representation only from the last one. In a similar fashion, the averaging or summing is optimized in the works of the Siamese CBOW model [47], Sent2 Vec model [48], and in other works [49; 50; 51]. A model derived from BERT, specially designed for sentence embeddings, SBERT, also uses averaging to pool words from two sequences to a pair of fixed-size sentence embeddings. Later, using the pair and its element-wise difference, the representation is passed to the softmax layer for classification and regression tasks. The authors of [52] tried to improve this configuration by replacing the averaging with a convolutional neural network. However, their improvement was limited to only better scores for the SALBERT model, which was still lower than SBERT based on averaging. Therefore, calculating the mean embedding from multiple word vectors is still a solid approach to the inner workings of current sentence models. Averaging is used in various scenarios. Some authors employ it to derive static embeddings from contextualized models. In [49], word vectors are learned by predicting them from the average of all contextual embeddings of words (except the target word) returned by the BERT encoder. Other authors of [53] derived static word embeddings from contextualized transformer representations by averaging word embeddings for each word in 100k different contexts. Such derived static versions are of better quality than classic Word2 Vec and share their benefits, such as having tens of millions of times lower computational cost than using standard contextual embedding models [49]. The other scenario is to compose a new more expressive meta-embedding from different models and domains. As a means of composition, the authors of [54] proposed concatenation with some trainable methods. However, a later work [55] found that averaging word vectors and padding them with zeros to compensate for dimensionality mismatches is a surprisingly effective method. Despite its popularity, averaging has inherent deficiencies. The most prominent is the loss of word order information.