# Towards Robust Word Embeddings for Noisy Texts
## 2 Towards Noise-Resistant Word Embeddings  
Word embedding models such as word2 vec, GloVe or fastText are able to cluster word variants together when given a big enough training corpus that includes standard and non-standard language [11]. That is, given enough examples where 'friend' (standard word), 'friend' (spell-checking error), 'frmd' (phonetic-compressed spelling) and even 'dog' or 'dawg' (street-talk) appear in similar contexts, these words will be translated to similar vector representations. Taking advantage of this fact, many state-of-the-art microtext normalization systems use word embeddings in their pipelines [14; 19; 20; 21], both when generating normalization candidates for the input words and also when selecting them.  
The problem with this approach is that the contexts where those example words appear are also likely to be affected by the same phenomena as the words themselves. For example, 'friend' might appear in phrases such as 'that's my best friend' or 'friend for life', while 'frmd' in others such as 'dats my bst frnd' or 'frmd 4 life'. This can make it difficult for the embedding algorithm to find the semantic similarity between 'friend' and 'frmd' when only relying on the assumption that the training corpus is big and diverse enough to effectively convey this variability. However, not all of the embedding algorithms are equally affected by this, as those which take subword information into account may have an advantage: in our example, the similar morphology shared by the word variants may be exploited by algorithms such as fastText, which uses character n-grams to give them more similar vector representations.  
In this paper, we present a modification of the skipgram model proposed by [PERSON] et al. [10] (a modification of the original by [PERSON] et al. [8]), which tries to improve the clustering of standard words and their noisy variants. This is attained through the use of _bridge-words_, normalized derivatives of the original words from the training corpus where one of their constituent characters is removed. By using these new words at training time in addition to the original ones, our objective is to increase the similarity between word variants, using those bridge-words as intermediate terms that match the words we want to cluster together. For example, 'friend' and 'freind' have in common the bridge-words 'frmd' and 'frend'. Even if the original words do not appear in the same context in the training corpus, using the bridge-words in place of the originals allows for indirect paths to be discovered: 'friend'-'frmd'-'freind' and 'friend'-'freind'. In the case of 'friend' and 'frmd', and assuming that we use an embedding algorithm that exploits subword information, as we propose here, the higher morphological similarities of the latter with respect to the bridge-words 'frend' and 'frmd' benefits their grouping together in the same cluster. As a side note, in the sense that these are intermediate (or normalized) representations that tie together otherwise isolated terms, they may resemble the index terms used in information retrieval. Since there is no index in our case, we will not refer to them as such. Notably, it should also be possible to apply analogous modifications to the ones described here to other training models, such as the continuous bag of words [8].  
It is worth pointing out that we did not consider the latest state-of-the-art models such as BERT [22] and GPT-2 [23] as it would not be feasible to apply analogous modifications to these large and complex models at this point, and GPT-3 [24] is out of the question in this regard. On the other hand, although we currently consider a monolingual English setup, our method should be suitable for any other language with a similar concept of _character_, in contrast to those based on logograms such as Chinese.