# Extracting Sentence Embeddings from Pretrained Transformer Models
## 5 Conclusions
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{2}{c}{**Last**} & \multicolumn{2}{c}{**First + Last**} & \multicolumn{2}{c}{**-**} \\ \hline avg. & 64.4 & 64.2 & 63.8 & 63.1 & 60.3 & 60.2 & 61.2 & 53.1 \\ + normalize & 64.4 & 64.2 & 63.8 & 63.1 & 60.3 & 60.2 & 61.2 & 53.1 \\ + quantile-u & 66.1 & 66.3 & 64.9 & 64.7 & 62.5 & 61.0 & 61.8 & 54.8 \\ + quantile-u\({}^{W}\) & 64.9 & 64.7 & 64.3 & 64.1 & 61.6 & 59.7 & 61.0 & 53.1 \\ + whiten\({}^{W}\) & 61.0 & 60.8 & 60.4 & 59.1 & 55.1 & 55.3 & 55.4 & 53.3 \\ + whiten\({}^{W}\) & 63.1 & 63.2 & 63.7 & 63.9 & 61.5 & 61.0 & 59.9 & 58.4 \\ + zscore & 66.3 & 66.7 & 65.0 & 64.9 & 62.8 & 62.1 & 63.1 & 56.3 \\ \hline id\({}^{W}_{t}\) & 62.5 & 62.0 & 61.7 & 59.7 & 57.5 & 61.9 & 60.1 & 57.4 \\ + normalize & 62.5 & 62.0 & 61.7 & 59.7 & 57.5 & 61.9 & 60.2 & 57.4 \\ + quantile-u & 63.0 & 62.6 & 61.8 & 60.4 & 58.5 & 60.8 & 59.4 & 52.9 \\ + quantile-u\({}^{W}\) & 62.9 & 62.5 & 61.8 & 60.5 & 59.0 & 61.1 & 59.5 & 54.7 \\ + whiten & 57.7 & 57.5 & 57.1 & 55.9 & 53.8 & 54.5 & 54.7 & 52.0 \\ + whiten\({}^{W}\) & 60.9 & 61.1 & 61.9 & 60.8 & 59.0 & 61.2 & 60.1 & 56.7 \\ + zscore & 63.7 & 63.4 & 63.2 & 62.3 & 60.9 & **62.9** & 61.3 & 57.3 \\ \hline id\({}^{T}_{t}\) & 64.4 & 65.0 & 62.8 & 61.1 & 59.2 & 60.2 & 61.3 & 56.8 \\ + normalize & 64.4 & 65.0 & 62.8 & 61.1 & 59.2 & 60.2 & 61.2 & 56.8 \\ + quantile-u\({}^{W}\) & 66.0 & 66.3 & 63.2 & 61.6 & 59.7 & 61.0 & 61.8 & 54.3 \\ + quantile-u\({}^{W}\) & 65.0 & 65.6 & 63.0 & 61.3 & 59.5 & 59.7 & 61.0 & 55.2 \\ + whiten & 59.9 & 60.3 & 58.1 & 56.8 & 54.4 & 55.3 & 55.4 & 52.5 \\ + whiten\({}^{W}\) & 63.5 & 64.2 & 62.4 & 61.0 & 58.9 & 61.0 & 59.9 & 56.3 \\ + zscore & 66.2 & 66.5 & 63.7 & 62.4 & 60.8 & 62.1 & **63.1** & 57.0 \\ \hline -biases & 65.1 & 65.9 & 65.6 & 65.1 & 63.8 & 59.0 & 59.7 & 59.7 \\ + normalize & 65.1 & 65.9 & 65.6 & 65.1 & 63.8 & 59.0 & 59.7 & 59.7 \\ + quantile-u & 66.9 & 67.7 & 66.0 & 65.7 & 64.7 & 59.9 & 60.2 & 56.5 \\ + quantile-u\({}^{W}\) & 65.6 & 66.5 & 65.8 & 65.5 & 64.4 & 58.6 & 59.3 & 57.3 \\ + whiten\({}^{W}\) & 64.8 & 65.4 & 64.7 & 63.9 & 61.3 & 59.8 & 58.6 & 58.0 \\ + whiten & 61.3 & 61.1 & 60.3 & 58.9 & 55.4 & 54.2 & 54.1 & 53.1 \\ + zscore & 67.1 & **67.8** & **66.3** & **66.3** & **65.6** & 61.1 & 62.0 & **60.4** \\ \hline
[MASK] & 64.8 & 65.0 & & & & & & \\ + normalize & 64.8 & 65.0 & & & & & \\ + quantile-u & 67.1 & 66.0 & & & & & \\ + quantile-u\({}^{W}\) & 65.6 & 65.6 & & & & & \\ + whiten & 61.7 & 61.8 & & & & & \\ + whiten\({}^{W}\) & 63.3 & 64.2 & & & & & \\ + zscore & **67.1** & 66.2 & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: _Cont._ \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{3}{c}{**Last**} & \multicolumn{3}{c}{**First + Last**} & \multicolumn{3}{c}{**-**} \\ \hline avg. & 85.4 & 80.7 & 81.2 & 84.4 & 79.0 & 85.0 & 85.0 & 28.0 \\ + normalize & 85.5 & 80.6 & 85.6 & 86.1 & 79.6 & **85.7** & 85.9 & 27.2 \\ + quantile-u & 85.2 & 86.9 & 84.0 & 85.1 & 79.7 & 85.2 & 85.2 & 27.5 \\ + quantile-u & 85.4 & 86.7 & 85.3 & 85.7 & 80.6 & 85.4 & 85.4 & 27.2 \\ + whiten & 34.3 & 31.3 & 31.3 & 31.5 & 30.5 & 29.9 & 29.9 & 28.8 \\ + whiten & 75.2 & 75.0 & 72.7 & 72.8 & 58.6 & 70.6 & 69.0 & 28.4 \\ + zscore & 86.0 & 81.2 & 81.3 & 83.8 & 79.8 & 84.8 & 85.2 & 27.7 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Spearman correlation dependence on token aggregation and post-processing techniques for semantic textual similarity STR task. The best result for each model is bolded, while underlined result is the best across all the models.