# Extracting Sentence Embeddings from Pretrained Transformer Models
## 4 Results
### Using Prompts  
The normal use of prompts, as presented in [15], is to place the sentence [X] inside a template as \"This sentence: \"[X]\" means [MASK]\" and use only the vector of [MASK] token from the final layer as the full representation of [X]. Our experiments show that in general, using prompts is beneficial; however, we found some ways to improve performance even more.  
First, extracting only the vector of [MASK] is not necessarily the best. We found that a simple average of all tokens, including the ones from the prompt template, and also the [MASK] token, is still a valid approach. Even more, it outperforms only the [MASK] token approach for clustering and classification tasks, with corresponding 55.0 and 79.9 average accuracies versus 54.2 and 76.8 for using only the [MASK] token (see T4 model results in Table 4).  
The use of the additional text template around the text enriches the target text representation. Let us compare the performance of 12 th layer averaged token representations with and without a template (see Figure 4, T0 avg. and T4 avg. with a template versus BERT avg. without). For all 3 groups of tasks--semantic textual similarity, clustering, and classification--the T0 template achieves 61.3 average Spearman correlation, 53.5 clustering accuracy, and 80.5 classification accuracy; the T4 template reaches 62.3, 55.0, and 79.9, while a regular, non-templated text obtains just 53.2, 50.6, and 79.0, respectively. This shows that the use of the templates allows the model to enrich target text representation. However,  
Figure 3: Alignment and uniformity of representations in relation to various token pooling and post-processing techniques. Lower values are better.  
this effect peaks at the last, 12 th layer, and the achieved performance is similar to the first + last layer combination of the regular non-templated vectors.  
Performance of using only the [MASK] token also peaks at the 12 th layer, so we investigated what influence it has in the enrichment of templated target text representation. Could it be that this token is the main culprit for better performance of averaged templated text representations? To answer this question, we tried to omit the [MASK] token from averaging (see Figure 4 T0 avg, no [MASK] and T4 avg. no [MASK]). For the T4 template and 12 th layer representations, averaging all tokens yields 62.3 average Spearman correlation, 55.0 clustering accuracy, and 79.9 classification accuracy; dropping the [MASK] token from averaging yields 58.7, 53.3, and 79.8; non-templated performance is 53.2, 50.6, and 79.0, respectively. We see that omitting the [MASK] token in averaging indeed hurts the performance. On the other hand, the results show that it is responsible only for approximately half of the improvements, with the other half coming from the other tokens in the templated text.  
One good reason to use averaging instead of only the [MASK] token is that then token weighting can be also applied. As we already showed in Table 4), the T4 template together with \(idf_{1}^{T}\) token weighting and uniform quantile post-processing allowed us to reach the  
Figure 4: Layer-wise performance of templated models T0 (**a**,**d**,**g**) and T4 (**b**,**e**,**h**), as well as BERT versus RE with no weighting or post-processing (**c**,**f**,**i**). The average performance of STS (**a**–**c**), clustering (**d**–**f**), and classification tasks (**g**–**i**) is shown by the lines, while shadow areas correspond to the standard deviation. We also show first + last aggregation over layers as the last tick \(\frac{1}{12}\) on the horizontal axis.  
average Spearman correlation of 71.6, which was the best among the tried methods. Also, we observed that post-processing on text representations from [MASK] token was not as effective as from averaged ones. This also suggests that all tokens in templated texts have richer representations.