# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Composing Word Vectors
#### 2.1.4 Weighted Average  
Not all constituent words are of the same importance; therefore, the corresponding vectors should be weighted. As a surprising event in Information Theory has higher information content than an expected event [57], some specific words should also add more weight to the sentence vector than others. In an early work [27], we can already find different weights for the members of the pairs of adjective-noun, noun-noun, or verb-object phrases. Later, work went from formal semantics to statistics-based measures, in particular the tf-idf scores. They can be used as features on their own, such as in [58], but it is better to use them to weight neural word vectors. For example, the authors of [59] proposed the Composite Document Vector as a concatenation of idf-graded weighted Word2 Vec vectors and tf-idf features. The authors of [60] proposed another better weighting approach, called Smooth Inverse Frequency (SIF). Here, higher-frequency words are down-weighted smoothly. The authors of [43] evaluated various compositional models and found that weighted averaging, in particular SIF, resulted in better performance in unsupervised similarity tasks that outperformed all other models. The author of [61] presented the uSIF method, an improvement to SIF, omitting hyperparameter tuning and constructing weighting with both word frequency and word vector length information. The authors of [62] also showed that the mean pooled output of transformers using tf-idf weights is better for clustering than the only regularly averaged output. A drastic case of weighting--a total removal of some words--is also shown to improve results. The authors of an older work [37] removed the stop words from the sequence before averaging. Meanwhile, as shown in [15; 63], the current performance of the BERT transformer model is also significantly improved in Semantic Text Similarity (STS) tasks if the most frequent tokens are removed. In other cases, the task itself focuses only on a few words, and others become redundant. The authors of [64] found that for the relation extraction task, some sentence embedding methods work better with shorter spans of words than the original sentences. They performed sentence segmentation in a way that a (sub)sentence would cover the identified entity mentions. Other authors of [64] also noted that SentenceBERT and Quickthought on spans or short segments containing two entity mentions are more clusterable than on the original sentences. These works also highlight the unequal contribution that each word has to the final sentence representation vector. Some works use more complex word weighting schemes. For example, the authors of [46] proposed a form of attention to weight each hidden state of LSTM. Other authors of [65] proposed the idea that each word brings a novel orthogonal basis to the sentence. Therefore, the length of a projection in this direction can be converted into a word's weight for use in the averaging. They come up with the final weight, consisting of three scores: novelty, significance, and corpus-wise uniqueness. Authors of [64] found that for the methods analyzed, such weighting of GloVe embeddings made them the most clusterable. Similar work by [66] incorporated alignment and novelty scores and applied them to contextualized representations of SBERT. Here, the alignment measures how well the word aligns with the neighboring ones, i.e., a well-aligned one is less informative and should be weighted less. The novelty, similar to [65], is expressed as the magnitude of the orthogonal component of the word to the subspace of neighboring words. Furthermore, the authors of [66] perform average weighting through all SBERT layers and also weight each layer-aggregated vector via \(l_{1}\)-normalized variance. This means that words that evolve faster across layers will receive higher weights since they have greater variance. Although complex to implement, such methods are reported to provide some minor performance improvements.