# Extracting Sentence Embeddings from Pretrained Transformer Models
## 4 Results
### Token Aggregation and Post-Processing Techniques  
The results of best-performing token aggregation and embedding post-processing techniques are presented in Table 4. The results here are averaged over all the different tasks in the class. See Appendix A for the individual results. We see that both aggregation and post-processing methods have a great impact on STS and clustering performance. For semantic textual similarity tasks, the best model was improved from 62.3 to 71.6 average Spearman correlation, while for clustering, the best model was improved from 59.2 to 64.8 average accuracy. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{3}{c}{**Last**} & \multicolumn{3}{c}{**First + Last**} & \multicolumn{3}{c}{**-**} \\ \hline Avg. Spearman correlation (\%) of STS tasks & & & & & & & \\ avg. & 61.3 & 62.3 & 61.3 & 61.4 & 56.4 & 56.7 & 60.6 & 50.3 \\ idf\({}^{W}_{t}\) & 67.8 & 67.4 & 69.2 & 69.3 & 66.9 & 66.1 & 65.4 & **66.4** \\ + zscore & 68.6 & 68.1 & **69.8** & **70.6** & **69.5** & **66.4** & **65.9** & 66.4 \\ idf\({}^{T}_{t}\) & 67.4 & 69.2 & 68.7 & 69.0 & 67.5 & 56.7 & 60.6 & 64.9 \\ + quantile-u & **69.5** & **71.6** & 68.9 & 69.5 & 68.4 & 56.2 & 61.0 & 63.0 \\ -biases & 64.7 & 66.0 & 68.2 & 69.3 & 67.3 & 54.1 & 57.5 & 63.7 \\ [MASK] & 63.4 & 69.3 & & & & & & \\ + quantile-u & 65.6 & 70.5 & & & & & & \\ SimCSE performance: 81.5 & & & & & & & & \\ \hline Avg. accuracy (\%) of clustering tasks & & & & & & & \\ avg. & 53.5 & 55.0 & 57.0 & 59.2 & 55.2 & 53.8 & 55.3 & 36.3 \\ idf\({}^{W}_{t}\) & 53.0 & 53.0 & 55.2 & 57.1 & 53.7 & 53.0 & 53.7 & 44.3 \\ + normalize & 54.3 & 54.4 & 58.0 & 57.9 & 54.1 & **55.8** & 56.1 & **49.5** \\ idf\({}^{T}_{t}\) & 53.1 & 57.6 & 60.7 & 62.5 & 58.7 & 53.8 & 55.3 & 42.8 \\ + quantile-u\({}^{W}\) & **57.6** & **60.0** & 63.1 & 64.4 & 60.2 & 54.6 & 56.0 & 44.4 \\ + normalize & 55.5 & 57.6 & **63.4** & **64.8** & 59.8 & 55.5 & **57.2** & 47.0 \\ -biases & 54.3 & 56.2 & 61.1 & 62.2 & 61.6 & 52.6 & 54.1 & 40.7 \\ + normalize & 54.4 & 56.3 & 62.4 & 63.4 & **62.2** & 54.4 & 56.8 & 44.6 \\ [MASK] & 45.5 & 54.2 & & & & & & \\ + quantile-u & 44.4 & 54.9 & & & & & & \\ \hline SimCSE performance: 59.8 and 64.0 (avg. + quantile-u\({}^{W}\), first + last layers) & & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Effect of different token aggregation and post-processing methods on multiple text embedding models. Both correlation and accuracy scores range from 0 to 100 (from the worst to the best). We also added results for SimCSE-BERT model [143], which is fine-tuned on NLI data supervision. The best result for each model is bolded, while underlined results are the best across all the models. Our techniques can even improve the dedicated SimCSE model [143], which was fine-tuned on NLI data. Its main strength lies in semantic textual similarity tasks, where it leads with over 10% difference. However, for clustering tasks, its average accuracy is similar to the other evaluated models at 59.8% and improves up to 64.0%, if we apply the best-performing techniques. This showcases a general tendency that the top-performing models are very good only in a narrow subset of tasks and highlights the importance of our more general methods. For maximum performance, usually, both aggregation and post-processing techniques must be used. As an example, sentences placed within the T4 template with no techniques applied have a 62.3 average Spearman correlation. With a better aggregation method of \(idf_{t}^{T}\), it is pushed up to 69.2, while only using quantile-u post-processing gives 67.3 (see Table 5 with only post-processing techniques applied). However, a combined effort of both \(idf_{t}^{T}\) and quantile-u gives the average Spearman correlation score of 71.6. Still, the improvements from both aggregation and post-processing do not exactly add up linearly.