# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Criticism
#### 2.2.5 Similarity Measures  
Some most basic tasks for sentences require measuring the distances between the corresponding embeddings. Usually, cosine similarity is employed. However, if the continuous representation space is curved or a text sequence is treated as a set of words, different metrics or representation space manipulations can be beneficial. The authors of [153] proposed Mutual Information (MI), well known in information theory and statistics, as a candidate for a similarity measure. They managed to successfully estimate MI for continuous random variables by the use of Kraskov-Stogbauer-Grassberger (KSG) estimator [154], which is based on elementary nearest-neighbor statistics. The other authors of [155] advised comparing sentence embeddings consisting of word vectors averaged by rank correlation, such as [PERSON]'s \(\tau\). They argued that such measures enable mean pooled representations to rival modern deep ones, used with cosine similarity. The authors of [156] proposed treating sentences as fuzzy sets of words and showed good performance with the specially adapted DynaMax similarity measure. They showed that word vectors alone are sufficient to achieve excellent performance on semantic textual similarity tasks when sentence embeddings and similarity measures are derived using ideas from fuzzy set theory. Other authors of [157] suggest processing each sentence with respect to its found \(k\)-nearest neighbors. Then, they find an optimal Euclidean subspace of the sentence manifold where cosine similarity would work best. The authors of [158] also project the sentences onto a fixed-dimensional manifold with the objective of preserving local neighborhoods in the original space. All these works mentioned employ various strategies to enhance the comparison of document representations, tailoring their approaches to the given task.  
Many similarity measures between sets employing earth mover's distance were proposed. The authors of [159] were the first to frame the similarity of documents as a transportation problem. The idea is that the distances between similar but different words in two documents should be small. Their Word Mover's Distance (WMD) is the cost of transporting a set of word vectors in an embedding space. This approach was effective, as it managed to exploit similarities between different Word2 Vec word vectors, such as the relation of analogies. However, the main deficiency of WMD was that the distances were expensive to compute. Moreover, an output, the single number of a distance between two given documents, can only be combined with \(k\)-nearest neighbors or \(k\)-means, while applications usually require a whole feature vector. The authors of a subsequent work [160] managed to derive vectors for documents using WMD. They constructed a positive-definite word mover's kernel using a feature map given by the WMD to random documents and then derived document embedding via a Random Features approximation of the kernel. Other authors of [70] proposed Word Rotator's Distance (WRD). It is designed so that the norm and angle of word vectors correspond to the probability mass and transportation cost in the earth mover's distance, respectively. Therefore, the norm of vectors, which is associated with the vector's significance, does not interfere with the calculation of transportation cost, as in WMD. Finally, the authors of [7 ] attempted to include structural information absent in the WMD's distance estimation between two sets of words. They represented the sentence vector as a weighted average of substructure vectors at a lower level in a recursive way, while a transport plan at a different level explains how the different substructures align.