# Towards Robust Word Embeddings for Noisy Texts
## 4 Related Work  
Word embeddings have been at the forefront of NLP research for the past decade, although the first application of vector representation of words dates back to [41]. More recently, the first models to attain wide use were word2 vec [8] and GloVe [9], which take words as basic and indivisible units, implying that the word vocabulary is fixed at training time and any unknown word would be given the same vector representation, regardless of its context or any other intrinsic property. To address the limitations of word2 vec and GloVe with out-of-vocabulary words, where morphologically-rich languages such as Finnish or Turkish are specially affected, new models appeared which take subword information into account. The type of subword information used varies in each particular approach: some of them require a preprocessing step to extract morphemes [42], while others employ a less strict approach by directly using the characters [43, 44] or character n-grams [10, 45] that form the words. When targeting noisy texts from social media, such as tweets from Twitter, previous work relies solely on the high coverage that can be obtained from training in an equally noisy domain [11]. An exception to this rule is the work from [PERSON] et al. [46], where they try to obtain robust embeddings to misspelled words (one or two edit operations away from the correct form) by using a new neural-based model. In this case, the flexibility is obtained by an encoding of the prefix, suffix and set of characters that form each word. By using this set of characters in the encoding, where the specific order between them is disregarded, this approach achieves some form of robustness to low-level noise, while the prefix and suffix part encodes most of the semantic information. The main difference of our approach is that we are not proposing a whole new model but a generic technique to adapt existing ones. This could be applied to many others, including that from [PERSON] et al. [46] itself. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **SE13 B** & **SE14 B** & **SE16 BD** & **SE16 CE** \\ \hline word2 vec & 84.3 & 88.3 & 77.4 & 35.1 \\ fastText & 83.3 & 88.1 & 76.5 & 33.7 \\ ours & **84.8** & **88.6** & **78.4** & **35.5** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy results of the extrinsic evaluation on SemEval (SE) Twitter SA datasets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **Standard** & \multicolumn{3}{c}{**Noisy**} & \multirow{2}{*}{**Standard**} & \multicolumn{3}{c}{**Noisy**} \\  & & _Low_ & _Mid_ & _High_ & & _Low_ & _Mid_ & _High_ \\ \hline \multicolumn{8}{c}{**Semantic sim. \& rel.**} & \multicolumn{3}{c}{**Binary classification**} \\ \hline word2 vec & 56.8 & 40.1 & 29.2 & 26.0 & 81.5 & 78.8 & 76.1 & 71.4 \\ fastText & **60.7** & 44.6 & 35.8 & 32.0 & **81.8** & 79.0 & 75.8 & 72.1 \\ ours & 60.4 & **48.1** & **41.5** & **37.7** & 81.6 & **79.4** & **77.7** & **74.1** \\ \hline \multicolumn{8}{c}{**Sentiment analysis**} & \multicolumn{3}{c}{**Entailment**} \\ \hline word2 vec & 57.9 & 55.4 & 50.5 & 42.8 & **66.3** & 54.9 & 48.8 & 35.8 \\ fastText & 58.8 & 55.8 & 52.6 & 47.0 & 66.2 & 54.0 & 50.0 & 40.2 \\ ours & **59.3** & **56.9** & **54.6** & **51.1** & **66.3** & **55.1** & **51.4** & **48.1** \\ \hline \multicolumn{8}{c}{**Question-type classification**} & \multicolumn{3}{c}{**Paraphrase detection**} \\ \hline word2 vec & **79.4** & 65.6 & 53.3 & 35.0 & 72.6 & **67.0** & **61.2** & **65.5** \\ fastText & 74.8 & 62.5 & 52.1 & 41.8 & 72.3 & 62.7 & 57.1 & 56.8 \\ ours & 73.4 & **67.5** & **59.4** & **49.5** & **72.9** & 66.9 & 60.2 & 56.3 \\ \hline \multicolumn{8}{c}{**Probing tasks**} & \multicolumn{3}{c}{} \\ \hline word2 vec & **58.2** & 51.5 & 45.3 & 39.3 & & & \\ fastText & 57.8 & 51.2 & 45.9 & 41.0 & & & \\ ours & 57.7 & **52.8** & **48.4** & **43.6** & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of the extrinsic evaluation on the SentEval benchmark. The noise levels are _low_ (\(p_{d}=0.3\)), _mid_ (\(p_{d}=0.6\)), and _high_ (\(p_{d}=1\)).