# Towards Robust Word Embeddings for Noisy Texts
## 3 Evaluation
### Results
#### 3.5.2 Extrinsic Evaluation  
Given the considerable amount of tasks and datasets included in the SentEval benchmark, we decided to group similar tasks and datasets and show the aggregated results from each group instead of following an exhaustive approach. In this case, and given the variability in dataset sizes, we use a weighted average as the aggregation function. First of all, we show in Figure 2 the dynamic behavior of each model when going from standard texts to noisier ones. In this case, we divided the tasks into two groups based on the performance metric: Spearman correlation or accuracy. The first one encompasses the semantic similarity and relatedness tasks (all STS* and SICK-Relatedness) and the second one the rest of them. Except in the case of word2 vec on the first group (yellow lines and crosses), all the models start from a very similar position in the standard scenario. Then, the performance begins its downward trend, where our model starts to stand out above the baselines. As we go towards noisier texts, our model manages to stay above the rest of the lines, increasing the distance margin up until the last stretch. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{3}{c}{**Standard**} & \\  & **SCWS** & **WS353** & **SL999** & **Sem17** \\ \hline word2 vec & 64.7 & 69.1 & 32.2 & 68.2 \\ fastText & **65.4** & 72.7 & 33.5 & 70.3 \\ ours & 65.1 & **73.1** & **33.8** & **70.4** \\ \hline \hline  & \multicolumn{3}{c}{**Noisy**} & \\  & **SCWS** & **WS353** & **SL999** & **Sem17** \\ \hline word2 vec & 13.0 & 13.7 & \(-\)10.9 & 11.2 \\ fastText & 35.2 & 38.1 & 7.3 & 37.2 \\ ours & **42.1** & **44.2** & **16.4** & **43.1** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Spearman correlation results of word similarity on SCWS, wordsim353 (WS353), SimLex999 (SL999), and SemEval17 (Sem17) datasets. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**Standard**} & \multicolumn{2}{c}{**Noisy**} \\  & **8-8-8** & **wiki** & **8-8-8** & **wiki** \\ \hline word2 vec & 59.4 & **53.8** & 22.8 & 39.3 \\ fastText & 65.6 & 49.0 & 31.7 & **41.1** \\ ours & **67.2** & 47.8 & **33.3** & **41.1** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Accuracy results of outlier detection on 8-8-8 and wiki-sem-500 (wiki) datasets. Next, Table 3 shows in greater detail the performance of each model in a less aggregated view. In this case, datasets have been grouped by task as described in Section 3.3. As we can see, our model is on par with the baselines on standard texts, with a few interesting exceptions: (1) it is able to obtain some advantage on sentiment analysis, which fastText also obtains over word2 vec; (2) on question-type classification, word2 vec obtains the best performance, and still clearly outperforms fastText on the lowest noise level, although not our model; and (3) on the probing tasks, word2 vec takes the lead again, this time by a smaller margin. Regarding noisy texts, our model is clearly superior on semantic similarity and relatedness, as we had already seen before, and it also outperforms the baselines on the rest of the tasks, with wider margins on noisier texts, but with the sole exception of paraphrase detection. In this surprising case, word2 vec outperforms both fastText and our model obtaining better accuracy on texts with the highest level of noise compared to the previous step. It appears that, with the proper training (and hence, vocabulary), word2 vec remains a strong baseline on extrinsic tasks, even in the case of noisy texts, where the level of noise has to be increased notably in order for fastText to obtain a clear advantage. This can also be observed following the continuous lines in Figure 2. On the other hand, the weakness seen on word semantic similarity (Table 1) relating to out-of-vocabulary words does not seem to translate to extrinsic tasks, where having more context and hence a higher chance of finding in-vocabulary words mitigates the problem, as we can see in the semantic similarity and relatedness (Table 3) results.