# Extracting Sentence Embeddings from Pretrained Transformer Models
## 4 Results
### Token Aggregation and Post-Processing Techniques
This indicates that the improvements to representations of this type may saturate below the perfect scores. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & **Last** & & & **First + Last** & & & \\ \hline Avg. accuracy (\%) of classification tasks & & & & & & & \\ avg. & 80.5 & 79.9 & 79.9 & 79.7 & 77.3 & 76.9 & 75.1 & 69.5 \\ idf\({}_{t}^{W}\) & 73.1 & 67.1 & 78.3 & 77.9 & 75.3 & 76.8 & 74.1 & 67.6 \\ idf\({}_{t}^{T}\) & 80.1 & 79.7 & 78.9 & 78.4 & 76.1 & 76.9 & **75.1** & 68.1 \\ -biases & 80.1 & **79.9** & **80.1** & **79.8** & 77.2 & 76.3 & 74.2 & **70.3** \\ [MASK] & 78.7 & 76.8 & & & & & & \\ SimCSE performance: 86.5 & & & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._ \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & **Last** & & & **First + Last** & & & \\ \hline Avg. Spearman correlation (\%) of STS tasks & & & & & & & \\ avg. & 61.3 & 62.3 & 61.3 & 61.4 & 56.4 & 56.7 & 60.6 & 50.3 \\ + zscore & **65.4** & **67.3** & **65.1** & **66.5** & 64.1 & 58.5 & **62.1** & 55.6 \\ + quantile-u & 65.0 & 67.3 & 64.3 & 65.7 & 63.0 & 56.2 & 61.0 & 53.0 \\ + quantile-u & 62.8 & 64.6 & 63.0 & 64.7 & 62.1 & 55.5 & 60.5 & 52.2 \\ + abtt-2 & 64.7 & 66.7 & 64.3 & 66.3 & 64.3 & 58.3 & 62.0 & 54.4 \\ + normalize & 61.3 & 62.3 & 61.3 & 61.4 & 56.4 & 56.7 & 60.6 & 50.3 \\ + whiten & 63.5 & 64.2 & 64.7 & 65.6 & 65.5 & **62.1** & 61.2 & 63.2 \\ + whiten\({}^{W}\) & 60.9 & 63.7 & 63.1 & 65.8 & **67.0** & 61.4 & 61.0 & **64.3** \\ \hline \hline \end{tabular}
\begin{tabular}{l c c c c c c c} \hline \hline Avg. accuracy (\%) of clustering tasks & & & & & & & \\ avg. & 53.5 & 55.0 & 57.0 & 59.2 & 55.2 & 53.8 & 55.3 & 36.3 \\ + zscore & 54.2 & 55.7 & 57.8 & 59.5 & 56.1 & 53.3 & 55.0 & 36.3 \\ + quantile-u & **54.6** & **56.3** & 58.7 & 60.7 & 57.3 & 54.2 & 55.8 & 36.8 \\ + quantile-u & 54.2 & 56.3 & **60.1** & **61.1** & **58.6** & 54.5 & 56.0 & 36.9 \\ + abtt-2 & 53.4 & 54.8 & 57.0 & 59.0 & 55.8 & 53.6 & 55.0 & 36.5 \\ + normalize & 53.4 & 55.0 & 59.7 & 59.7 & 56.1 & **55.5** & **57.2** & 38.5 \\ + whiten & 35.4 & 35.7 & 33.1 & 29.8 & 26.8 & 25.3 & 26.0 & 24.8 \\ + whiten\({}^{W}\) & 49.0 & 49.8 & 56.2 & 58.7 & 55.0 & 50.4 & 50.1 & **41.5** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance of post-processing with plain token averaging. abtt-2 is the ABTT method with top 2 principal components removed. The best result for each model is bolded, while underlined results are the best across all the models. We considered many post-processing techniques for sentence vectors and show the performance of the most popular and best-performing ones in Table 5 separately. Indeed, almost every method somehow improves the average Spearman correlation for STS tasks. For all-but-the-top (abtt), we varied the number of components to remove up to a hundred but settled on removing only the first two, as it was slightly better than the rest. Although highly credited in the literature, abtt-2 still obtained smaller scores than the others. Plainly averaging the token representations, the highest scores for both STS and clustering tasks were achieved using quantile-uniform normalization. We would also like to mention the simple normalization of vectors to the unit length, which was often beneficial for clustering tasks, and whitening normalization, which was learned on Wikitext-2 dataset, and was favorable with the random embeddings model. We trained other post-processing methods on Wikitext-2 too, yet they resulted in similar or slightly smaller scores. Unlike unsupervised ones, classification tasks do not benefit from the two representation-shaping techniques. There is only negligible improvement for classification tasks, when the biases (punctuation, most frequent, and subword tokens) are eliminated. Otherwise, performance is only decreased.