# Extracting Sentence Embeddings from Pretrained Transformer Models
## 5 Conclusions
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{3}{c}{**Last**} & \multicolumn{3}{c}{**First + Last**} & \multicolumn{3}{c}{**-**} \\ \hline idf\({}_{t}^{W}\) & 79.5 & 81.4 & 80.6 & 84.0 & 78.0 & 83.6 & 83.7 & 39.9 \\ + normalize & 84.9 & 85.6 & 84.7 & 85.1 & 78.3 & 85.0 & 84.9 & 41.4 \\ + quantile-u & 85.8 & 85.7 & 81.7 & 84.4 & 78.6 & 84.2 & 84.2 & 41.2 \\ + quantile-u\({}^{W}\) & 85.6 & 85.8 & 81.8 & 84.7 & 79.1 & 84.6 & 84.7 & **43.4** \\ + whiten & 30.9 & 32.0 & 31.3 & 31.8 & 30.7 & 30.7 & 30.1 & 27.6 \\ + whiten\({}^{W}\) & 74.5 & 78.1 & 73.8 & 74.4 & 70.9 & 72.5 & 72.1 & 38.4 \\ + zscore & 85.9 & 81.8 & 80.4 & 84.0 & 78.9 & 83.7 & 83.7 & 41.0 \\ \hline idf\({}_{t}^{W}\) & 80.1 & 82.1 & 80.9 & 82.5 & 79.3 & 85.0 & 85.0 & 28.7 \\ + normalize & 85.2 & 82.2 & 85.1 & 84.8 & 81.7 & 85.7 & **85.9** & 28.5 \\ + quantile-u & 85.9 & 87.0 & 82.4 & 81.9 & 80.9 & 85.2 & 85.2 & 28.6 \\ + quantile-u\({}^{W}\) & 85.9 & 87.0 & 82.0 & 84.4 & 82.1 & 85.3 & 85.5 & 28.4 \\ + whiten & 30.2 & 32.6 & 30.9 & 31.3 & 29.9 & 29.9 & 28.0 & 28.0 \\ + whiten\({}^{W}\) & 72.1 & 68.4 & 73.0 & 71.9 & 45.5 & 70.6 & 69.0 & 28.9 \\ + zscore & 85.9 & 82.6 & 81.1 & 82.1 & 79.7 & 84.8 & 85.2 & 28.8 \\ \hline -biases & 85.7 & 86.4 & 81.4 & 85.6 & 83.4 & 84.0 & 83.8 & 28.5 \\ + normalize & 85.7 & 86.6 & **86.8** & **86.8** & **83.8** & 85.3 & 85.5 & 28.4 \\ + quantile-u & 85.4 & 86.7 & 85.7 & 85.4 & 83.4 & 84.7 & 84.7 & 28.5 \\ + quantile-u\({}^{W}\) & 85.6 & 87.1 & 86.1 & 86.0 & 83.6 & 84.6 & 84.9 & 29.4 \\ + whiten\({}^{W}\) & 73.5 & 73.4 & 72.4 & 72.1 & 55.5 & 68.8 & 67.6 & 28.2 \\ + whiten & 31.1 & 31.5 & 33.0 & 30.4 & 30.3 & 29.5 & 29.8 & 28.3 \\ + zscore & **86.0** & **87.1** & 81.7 & 84.8 & 83.5 & 84.4 & 84.5 & 28.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: _Cont._ \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **T0** & **T4** & **BERT** & **BERT + Avg.** & **Avg.** & **B2S** & **B2S-100** & **RE** \\ \cline{2-9}
**Layer** & \multicolumn{2}{c}{**Last**} & \multicolumn{2}{c}{**First + Last**} & \multicolumn{2}{c}{**-**} \\ \hline avg. & 63.5 & 64.3 & 66.1 & 66.1 & 65.1 & 66.2 & 65.8 & 61.2 \\ + normalize & 62.7 & 64.6 & 66.2 & 67.1 & 66.2 & **68.3** & **67.7** & 67.1 \\ + quantile-u & 64.8 & 65.1 & 67.1 & 67.4 & 67.3 & 66.3 & 66.9 & 63.5 \\ + quantile-u\({}^{W}\) & 63.9 & 64.7 & 66.4 & 68.0 & 66.7 & 66.9 & 67.2 & 63.6 \\ + whiten & 59.2 & 60.3 & 57.5 & 57.3 & 54.5 & 58.1 & 58.4 & 53.1 \\ + whiten\({}^{W}\) & 62.4 & 62.0 & 65.6 & 66.1 & 65.2 & 65.8 & 65.8 & 61.7 \\ + zscore & 63.6 & 65.0 & 66.7 & 67.1 & 65.9 & 66.0 & 66.6 & 61.8 \\ \hline id\({}^{W}_{t}\) & 63.5 & 62.6 & 65.2 & 66.2 & 65.6 & 64.5 & 65.8 & 63.8 \\ + normalize & 64.3 & 63.1 & 66.0 & 66.5 & 66.0 & 66.5 & 66.7 & **69.5** \\ + quantile-u & 64.0 & 65.4 & 66.6 & 66.5 & 67.0 & 66.4 & 66.1 & 64.1 \\ + quantile-u\({}^{W}\) & 64.4 & 64.9 & 67.5 & 67.8 & 66.6 & 67.4 & 65.9 & 63.4 \\ + whiten & 57.9 & 59.9 & 58.3 & 56.5 & 55.2 & 58.9 & 58.6 & 51.4 \\ + whiten\({}^{W}\) & 61.9 & 61.4 & 66.2 & 67.7 & 66.7 & 65.5 & 66.7 & 63.3 \\ + zscore & 63.5 & 63.3 & 66.1 & 66.4 & 65.7 & 65.9 & 65.6 & 63.9 \\ \hline id\({}^{T}_{t}\) & 63.2 & 64.9 & 66.4 & 67.4 & 65.9 & 66.2 & 66.0 & 61.3 \\ + normalize & 64.7 & 64.7 & 67.7 & **68.6** & 65.9 & 68.3 & 67.7 & 67.4 \\ + quantile-u & **65.2** & 65.5 & **68.5** & 67.9 & 67.5 & 66.6 & 67.0 & 63.7 \\ + quantile-u\({}^{W}\) & 64.8 & **66.2** & 67.7 & 67.3 & 67.3 & 67.1 & 67.2 & 62.9 \\ + whiten & 58.8 & 59.2 & 58.7 & 59.0 & 55.6 & 57.7 & 58.5 & 55.1 \\ + whiten\({}^{W}\) & 60.1 & 60.4 & 65.1 & 66.7 & 64.8 & 65.8 & 65.8 & 61.2 \\ + zscore & 63.8 & 65.0 & 66.4 & 67.6 & 66.3 & 66.0 & 66.3 & 62.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: _Cont._