# Extracting Sentence Embeddings from Pretrained Transformer Models
## 5 Conclusions  
We empirically evaluated the effects of various aggregation and post-processing techniques of token representations in a trained transformer and other models to form good text-, paragraph-, or sentence-level embeddings. We carried out the empirical evaluation of the embeddings on three classes of downstream text-level tasks: Semantic Textual Similarity (STS), clustering, and classification. We found the techniques to benefit all models studied for the unsupervised STS (the best model average Spearman correlation increased from 62.3% to 71.6%) and clustering (the best model average clustering accuracy increased from 59.2% to 64.8%), while it had no positive effect on the supervised classification tasks (see Table 4). We present a strong and very simple baseline model of Random Embeddings (RE), where every token is assigned a random vector as its embedding. Combined with token aggregation and post-processing techniques, it also almost matches the average STS performance of the BERT model with the techniques applied, with 66.4% versus 69.8% average Spearman correlation. It also shows very high performance for some tasks, like stackoverflow classification, where BERT token contextualization may not work well on code samples in the texts (see Section 4.1.2 for more details). We encourage future work to use RE as a baseline, due to its mid-level performance, simple implementation, and ability to separate the contribution to the performance of the learned contexts from the aggregation and post-processing techniques. We found that the aggregation and post-processing techniques tried typically increase the isotropy of the representations, and the isotropy for most models is positively correlated (up to 69% Pearson correlation) with the Spearman correlation of STS tasks (Section 4.1.3). The highest isotropy improvement is observed for our Random Embeddings model, since its token representations have the maximal isotropy to start with. We did not find the token aggregation and post-processing techniques to improve the alignment and uniformity properties of representations. We question the use of prompts (adding a sentence into a certain text template) for retrieving the representation of the sentence from only the [MASK] token. Our experiments show that averaging all the templated text tokens, with idf weighting and post-processing for the unsupervised tasks, is better. Meanwhile, the average increase in performance due to the added template is only obvious for the STS tasks, gives no improvement in clustering, and is very slight in classification (see Table 4, Section 4.2). We presented a static vector model Avg., which simply contains BERT tokens averaged over multiple different contexts. Our experiments show that it outperforms a more complex BERT2 Static [49], also a static word-level model, yet specially trained on BERT contexts. With the best post-processing and token aggregation techniques, the advantage for unsupervised STS tasks is 69.5 versus 66.4, and for clustering, it is 62.2 versus 57.2., with a negligible difference for classification tasks. Moreover, we show that combining Avg. with the parent BERT model can bring even further improvements. In particular, BERT + Avg. reached the highest average clustering accuracy of 64.8 out of all our considered models, as well as the classification accuracy of 80.5 (Section 4.3). We encourage future work to also use Avg. as a baseline, both due to its upper-level performance and simple implementation. In our work, we also analyzed prompt and BERT + Avg. models layer-wise. We found that for the STS tasks, taking the representations from the first layers performs better, and for the clustering task, the performance profile forms a \"U\" shape with tops at the first and last layers. Therefore, for these two task groups, we mostly use the average of first + last layers, harnessing both of the tops for the best performance. On the other hand, classification tasks have an inverted \"U\" shape with the top in the 10 th layer (Figure 5).