# Extracting Sentence Embeddings from Pretrained Transformer Models
## 5 Conclusions
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{3}{c}{**Last**} & \multicolumn{3}{c}{**First + Last**} & \multicolumn{3}{c}{**-**} \\ \hline avg. & 70.2 & 70.6 & 70.5 & 70.0 & 64.2 & 69.8 & 74.2 & 62.1 \\ + normalize & 70.2 & 70.6 & 70.5 & 70.0 & 64.2 & 69.8 & 74.2 & 62.1 \\ + quantile-u & 71.8 & 73.7 & 72.7 & 74.3 & 72.4 & 68.9 & 73.6 & 61.4 \\ + quantile-u & 71.5 & 72.5 & 72.3 & 74.1 & 72.6 & 69.4 & 74.4 & 62.7 \\ + whiten & 62.6 & 62.9 & 64.0 & 65.9 & 69.3 & 69.3 & 68.8 & 67.9 \\ + whiten\({}^{\text{W}}\) & 70.2 & 71.8 & 71.9 & 74.9 & 77.0 & 74.7 & 75.2 & 74.7 \\ + zscore & 71.9 & 73.0 & 72.6 & 74.3 & 72.4 & 69.3 & 73.9 & 64.3 \\ \hline id\({}_{t}^{\text{W}}\) & 75.4 & 74.4 & 75.8 & 75.4 & 71.5 & 73.7 & 74.4 & 74.4 \\ + normalize & 75.4 & 74.4 & 75.8 & 75.4 & 71.5 & 73.7 & 74.4 & 74.4 \\ + quantile-u & 75.1 & 74.2 & 75.3 & 75.7 & 73.7 & 73.4 & 71.9 & 68.5 \\ + quantile-u\({}^{\text{W}}\) & **76.5** & 75.6 & 76.3 & 76.9 & 74.8 & 74.6 & 73.7 & 71.1 \\ + whiten & 62.7 & 61.4 & 64.3 & 65.2 & 66.3 & 66.8 & 66.7 & 65.0 \\ + whiten\({}^{\text{W}}\) & 72.2 & 71.6 & 75.0 & 76.5 & 76.6 & **75.4** & **75.5** & 74.4 \\ + zscore & 74.2 & 73.1 & 75.2 & 76.2 & 75.0 & 73.8 & 74.4 & 73.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Spearman correlation dependence on token aggregation and post-processing techniques for semantic textual similarity STS15 task. The best result for each model is bolded, while underlined result is the best across all the models. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{2}{c}{**Last**} & \multicolumn{2}{c}{**First + Last**} & \multicolumn{2}{c}{**-**} \\ \hline avg. & 69.1 & 69.7 & 67.9 & 65.2 & 56.4 & 55.2 & 58.5 & 55.5 \\ + normalize & 69.1 & 69.7 & 67.9 & 65.2 & 56.4 & 55.1 & 58.5 & 55.5 \\ + quantile-u & 70.1 & 72.4 & 70.3 & 71.0 & 67.0 & 55.9 & 59.5 & 54.8 \\ + quantile-u\({}^{W}\) & 69.7 & 71.5 & 69.8 & 70.0 & 65.3 & 54.6 & 57.9 & 55.1 \\ + whiten & 65.4 & 65.9 & 67.1 & 68.9 & 69.9 & 65.0 & 63.0 & 67.1 \\ + whiten\({}^{W}\) & 64.7 & 67.2 & 69.9 & 71.5 & 71.1 & 63.1 & 62.8 & 68.3 \\ + zscore & 69.3 & 71.0 & 72.0 & 73.2 & 69.8 & 60.2 & 62.3 & 60.4 \\ \hline id\({}^{W}_{t}\) & 70.1 & 69.1 & 73.9 & 73.0 & 69.5 & 68.8 & 68.0 & 71.9 \\ + normalize & 70.1 & 69.1 & 73.9 & 73.0 & 69.5 & 68.8 & 68.0 & 71.9 \\ + quantile-u & 70.5 & 69.9 & 73.7 & 74.2 & 72.9 & 68.1 & 65.8 & 68.9 \\ + quantile-u\({}^{W}\) & 70.5 & 69.8 & 74.0 & 74.4 & 72.7 & 67.9 & 66.2 & 69.6 \\ + whiten & 64.6 & 64.0 & 68.0 & 68.9 & 69.1 & 65.6 & 63.4 & 67.3 \\ + whiten\({}^{W}\) & 70.0 & 69.1 & 75.6 & 76.0 & **74.8** & 69.6 & 68.7 & **72.4** \\ + zscore & 72.2 & 71.7 & **75.6** & **76.4** & 74.8 & **70.3** & **69.5** & 72.2 \\ \hline id\({}^{T}_{t}\) & 72.9 & 73.1 & 72.8 & 72.0 & 69.7 & 55.2 & 58.5 & 69.1 \\ + normalize & 72.9 & 73.1 & 72.8 & 72.0 & 69.7 & 55.2 & 58.5 & 69.1 \\ + quantile-u & **73.9** & **75.0** & 72.8 & 73.0 & 71.8 & 55.9 & 59.5 & 67.3 \\ + quantile-u\({}^{W}\) & 73.7 & 74.6 & 72.7 & 72.7 & 71.2 & 54.6 & 57.9 & 67.5 \\ + whiten & 67.0 & 68.6 & 68.0 & 68.7 & 69.5 & 65.0 & 63.0 & 67.8 \\ + whiten\({}^{W}\) & 69.0 & 71.0 & 73.2 & 72.8 & 71.3 & 63.1 & 62.8 & 69.3 \\ + zscore & 73.1 & 73.1 & 74.4 & 74.4 & 73.1 & 60.2 & 62.3 & 70.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Spearman correlation dependence on token aggregation and post-processing techniques for semantic textual similarity STS16 task. The best result for each model is bolded, while underlined result is the best across all the models.