# Extracting Sentence Embeddings from Pretrained Transformer Models
###### Abstract  
Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT's hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT's sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models. BERT; embeddings; large language models; natural language processing; text embeddings; sentence vector representation; semantic similarity; transformer models; prompt engineering; unsupervised learning +
Footnote †: journal:  
the solving of more sophisticated tasks such as sentiment analysis or question answering, as well as became the state-of-the-art in almost every NLP task. Therefore, it should possess the representation of the whole text sequence. Transformer models generally have at least 12 layers and their parameters are counted in hundreds of millions or much more. Condensing the sentence representation using these weights into a fixed 768-length vector (a common length for token vectors in base transformer models) is a challenge. It turns out that simply extracting features from a transformer model's last layer activations yields even worse results than much simpler models [9]. There were multiple works that tried to optimize the architecture parameters (see [10], which evaluated multiple proposed modifications), and multiple investigative works probing the properties of different parts of the model (see a critical review [11] of such approaches). Only the following methods gave tangible results: (1) the plain embedding averaging of all tokens comprising the sequence; (2) engineering a prompt template to condensate sentence representation into a single token; and (3) using a specially dedicated fine-tuned model designed to produce such vectors. Fine-tuning is definitely the most efficient option. Resulting models, such as InferSent [12] or Sentence-BERT [9], showed state-of-the-art performance in sentence-level tasks at the time. However, the success of fine-tuning depends on several factors. It requires high-quantity and high-quality target domain data, as well as computational resources, which may not always be available. Sometimes, even the existing target domain data cannot be used, as they are very expensive to label. Other difficulties emerge if the data contain sensitive or private information and present a risk of it surfacing during inference. Having such constraints, one has to resort to the first two options of using raw encoded features to produce a vector for a text sequence. Using feature aggregation instead of fine-tuning also allows us to better explain the inner workings of the state-of-the-art black-box models.