# Extracting Sentence Embeddings from Pretrained Transformer Models
## 3 Methods
### Evaluation
#### 3.5.2 Semantic Textual Similarity (STS) Tasks
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Split**} & \multirow{2}{*}{**N**} & \multirow{2}{*}{**C**} & \multirow{2}{*}{**L/S**} & \multicolumn{2}{c}{\(\|V\|\)} & \multicolumn{2}{c}{**Len**} & \multicolumn{2}{c}{**Alpha, \%**} \\ \cline{5-10}  & & & & & **W** & **B** & **W** & **B** & **W** & **B** \\ \hline STS tasks & & & & & & & & & & \\ STS12 & 3108 & & 5.2 & 8127 & 7802 & 25 & 28 & 83 & 77 \\ STS13 & 1500 & & 0.7 & 5152 & 5141 & 20 & 21 & 88 & 82 \\ STS14 & 3750 & & 1.6 & 9117 & 8613 & 21 & 23 & 86 & 80 \\ STS15 & 3000 & & 0.9 & 7364 & 7185 & 23 & 24 & 89 & 85 \\ STS16 & 1186 & & 1.2 & 3971 & 4175 & 26 & 28 & 87 & 83 \\ STR & 5500 & & 1.0 & 22,392 & 12,883 & 25 & 32 & 83 & 76 \\ \hline Binary classification & & & & & & & & & \\ MR & 10,662 & 2 & 1.0 & 20,325 & 13,802 & 22 & 26 & 84 & 76 \\ CR & 3775 & 2 & 1.8 & 5675 & 5221 & 20 & 22 & 84 & 80 \\ SUBJ & 10,000 & 2 & 1.0 & 22,636 & 15,912 & 25 & 28 & 85 & 78 \\ MPQA & & 10,606 & 2 & 2.2 & 6239 & 6248 & 3 & 3 & 97 & 88 \\  & train & 67,349 & 2 & 1.3 & 14,816 & 11,570 & 9 & 11 & 88 & 78 \\ SST2 & dev & 872 & 2 & 1.0 & 4339 & 4542 & 20 & 23 & 85 & 76 \\  & test & 1821 & 2 & 1.0 & 7053 & 6824 & 19 & 23 & 85 & 76 \\ MRPC & train & 4076 & 2 & 2.1 & 16,112 & 12,061 & 44 & 50 & 81 & 75 \\  & test & 1725 & 2 & 2.0 & 10,092 & 8471 & 44 & 50 & 82 & 75 \\ \hline \multicolumn{10}{l}{Fine-grained classification} & & & & & & & & \\  & train & 8544 & 5 & 2.1 & 16,579 & 12,395 & 19 & 23 & 84 & 75 \\ SST5 & dev & 1101 & 5 & 2.1 & 5038 & 5168 & 19 & 23 & 85 & 76 \\  & test & 2210 & 5 & 2.3 & 7929 & 7478 & 19 & 23 & 85 & 76 \\ TREC & train & 5452 & 6 & 14.5 & 9437 & 8492 & 10 & 11 & 87 & 81 \\  & test & 500 & 6 & 15.3 & 1100 & 1247 & 7 & 8 & 85 & 79 \\  & train & 7320 & 3 & 4.4 & 27,775 & 13,603 & 31 & 40 & 100 & 77 \\ SCICITE & dev & 916 & 3 & 4.4 & 7625 & 6918 & 31 & 40 & 100 & 78 \\  & test & 1861 & 3 & 3.8 & 12,609 & 9217 & 31 & 41 & 100 & 77 \\  & train & 4500 & 3/... & 3.8/3.6 & 2258 & 2277 & 19 & 20 & 99 & 96 \\ SICK-E/R & dev & 500 & 3/... & 3.8/5.0 & 1122 & 1172 & 20 & 20 & 99 & 96 \\  & test & 4927 & 3/... & 3.9/3.9 & 2271 & 2291 & 19 & 20 & 99 & 96 \\  & train & 5749 &... & 1.3 & 12,430 & 10,792 & 22 & 25 & 86 & 80 \\ STS-B & dev & 1500 &... & 0.7 & 6542 & 6511 & 26 & 28 & 85 & 80 \\  & test & 1379 &... & 1.1 & 4888 & 4921 & 22 & 24 & 85 & 81 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Dataset statistics for STS and classification tasks. N is the number of text samples, C is the number of clusters, L/S is the imbalance number, defined as the size of the largest class divided by that of the smallest class (for STS tasks, the two classes are binned to 1 point label value length ones from the highest and lowest sides), \(\|V\|\) is vocabulary size, Len is the average number of tokens in each text sample, Alpha is the percent of tokens that are alphabetic. Statistics with plain word tokenization are marked with “W” and with bert-base-uncased model tokenizer as “B”. For MRPC, SICK-R/E, STS-B, and STS tasks, statistics are calculated for tokenized and then concatenated sentences in each pair. we concatenate all subsets and report the general Spearman correlation for that year (referred to in [143] as \"all\"). As we used the SentEval toolkit [194], we had to implement concatenation ourselves since it only had \"mean\" and \"wmean\" settings available.