# Extracting Sentence Embeddings from Pretrained Transformer Models
###### Abstract
Different parts of a transformer model may be responsible for different levels of representation, which favor different tasks. It is also important which levels in the representation hierarchy are easier to shape or process. A better understanding of the inner workings could help address hallucinations or other problems that current large consumer-grade language models face. One of the two pre-training tasks of the famous BERT model [5] was next sentence prediction. It was optimized through a special [CLS] token, which sought to capture the whole-sequence-level representation. But in later works, such as [9], it was revealed that such a representation is very poor, not better than the classic ones, and the authors opted for simple averaging of the last layer tokens instead. The authors of [13] proposed using averaging of tokens of the first and last layers, and the authors of [14] also included hidden token representations of the second layer. But is that really the best way to obtain a numerical representation of the text's meaning? Pre-trained models like BERT capture a lot of useful representations, yet it is not that trivial to extract them. The authors of [15] showed that further improvements can be achieved by removing the most frequent, sub-word, uppercase, and punctuation tokens before averaging. Furthermore, even larger gains can be achieved by using a prompt template, \"This sentence: \"[X]\" means [MASK]\", where the target sentence is placed instead of [X] and the representation of the token [MASK] is used as the final representation of the whole sentence. Such a method is a good example of representation extraction without any specific fine-tuning. Inspired by the above findings, we hypothesize that there may be more ways to distill relevant sentence-level embeddings without directly fine-tuning the pre-trained BERT model. More concretely, in our analysis, we try to find a function that would shape and extract representations of bert base-uncased along its layers, target tokens, and additional corpus, so that the best performance in multiple tasks would be achieved. We evaluate our approach on short text clustering, semantic textual similarity, and classification tasks. Considering the difficulties of acquiring representations from the state-of-the-art transformer models and following the success of recent works demonstrating it, our approach offers these main contributions:  
* We provide an extensive and organized review of related work on producing sentence-level embeddings from transformer models. * We experimentally test how multiple combinations of various of the most promising token aggregation and sentence representation post-processing techniques impact the performance of three classes of different tasks and properties of representations on several models. * We propose two competitive and simple static token models as baselines: random embeddings and averaged representations (\"Avg\"). * We propose an improvement for BERT: the BERT + Avg combined model. We experimentally test many weights and layers of how the representations of BERT and Avg can be most effectively mixed. The rest of this paper is organized as follows. We provide a review of related work in the literature on composing word vectors and representation reshaping to obtain sentence- or text-level embeddings in Section 2. In Section 3, we outline the experimental setting and give a detailed background on our chosen approach, models, and datasets. In Section 4, we present the results. Finally, we summarize the findings of this work in Section 5.