# Extracting Sentence Embeddings from Pretrained Transformer Models
## 3 Methods
### Evaluation
#### 3.5.2 Semantic Textual Similarity (STS) Tasks  
STS assesses the degree to which two sentences are semantically equivalent to each other. A single sample consists of two sentences and a score ranging from 0 for no meaning overlap to 5 for meaning equivalence. The semantic textual similarity shared task has been held annually since 2012 up to 2017 [216; 217; 218; 219; 220; 221] and formed STS12-STS17 datasets. A total of 8628 carefully collected samples from these contests formed the STS benchmark [221]. Table 3 shows details of the STS datasets, including SICK-Relatedness [222] and STS-B test sets, which we also use. Similarly to [145], we also add (STR) [223], a recent semantic relatedness dataset created by comparative annotations. \begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**N**} & \multirow{2}{*}{**C**} & \multirow{2}{*}{**L/S**} & \multicolumn{2}{c}{\(\|V\|\)} & \multicolumn{2}{c}{**Len**} & \multicolumn{2}{c}{**Alpha, \%**} \\ \cline{5-10}  & & & & \multicolumn{1}{c}{**W**} & \multicolumn{1}{c}{**B**} & \multicolumn{1}{c}{**W**} & \multicolumn{1}{c}{**B**} & \multicolumn{1}{c}{**W**} & \multicolumn{1}{c}{**B**} & \multicolumn{1}{c}{**W**} & \multicolumn{1}{c}{**B**} \\ \hline agnews & 8000 & 4 & 1 & 21,062 & 16,140 & 23 & 26 & 100 & 86 \\ biomedical & 20,000 & 20 & 1 & 18,888 & 9326 & 13 & 20 & 98 & 64 \\ googleTS & 11,109 & 152 & 143 & 19,508 & 14,763 & 28 & 33 & 100 & 85 \\ searchsnippets & 12,340 & 8 & 7 & 30,643 & 16,334 & 19 & 24 & 93 & 77 \\ stackoverflow & 20,000 & 20 & 1 & 22,909 & 7332 & 8 & 12 & 87 & 71 \\ tweet & 2472 & 89 & 249 & 5098 & 5091 & 9 & 11 & 100 & 81 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Dataset statistics for the short text clustering datasets. N is the number of text samples, C is the number of clusters, L/S is the imbalance number defined as the size of the largest class divided by that of the smallest class, \(\|V\|\) is vocabulary size, Len is the average number of tokens in each text sample, Alpha is the percent of tokens that are alphabetic (all token characters must be defined in the Unicode character database as “Letter”, while tokens with numbers, punctuation, or BERT continuation tokens such as “##ing” are excluded). Statistics with plain word tokenization are marked with “W” and with bert-base-uncased model tokenizer as “B”. STS is a very popular choice for evaluating textual embeddings in an unsupervised way. Without any fine-tuning, one can calculate the distance (usually cosine) between two vectors of two sentences, which should correlate to the target equivalence score. It is so widely adopted that almost all work on semantic representations assesses the model performance on this task. We follow the STS evaluation settings from [143]. First, the evaluation is kept unsupervised by not applying any additional regressors; cosine similarity between embeddings in a pair is taken directly as a model score for similarity. To find the degree of correlation between the annotated and model-given labels, [PERSON]'s rank correlation is used because it measures the rankings instead of the actual scores. Finally, for annual STS challenges,