# Towards Robust Word Embeddings for Noisy Texts
## 2 Towards Noise-Resistant Word Embeddings
### Modified Skipgram Model  
The skipgram model found in tools like word2 vec and fastText establishes that, for each word in a text, it should be possible to predict those in their corresponding contexts [8]. As a consequence, the words that appear in similar contexts end up represented by similar vectors, so that the transformation learned by the model can effectively map one group of words onto the other. Based on the skipgram model from fastText, our proposal aims at increasing the similarity between standard words and their noisy counterparts. To accomplish this, we introduce a new set of words at training time that we denominate _bridge-words_. For each word in the training corpus, we first put the words into lowercase, strip the accents and remove character successive repetitions, and then obtain one bridge-word for each remaining character in the word, by removing one different character each time. For character repetitions, we cover both standard and non-standard ones (e.g.,'success' vs 'daaammn'), obtaining a common denominator for when users make the mistake of removing standard repetitions (e.g., from'success' to'succes'), or add repetitions to provide emphasis (e.g., from 'damn' to 'daaammn'). The resulting words are very similar and can still be read mostly in the same way. An analogous reasoning is used in the case of lowercasing and stripping the accents. Note that this procedure is exclusively applied to obtain all the bridge-words, and the unprocessed corpus will be used during training. Formally, let \(\mathcal{V}\) be the word vocabulary extracted from the training corpus so that \(\mathcal{V}=\{w_{1},w_{2},...,w_{n}\}\) with \(n\) the size of the vocabulary. The set of bridge-words is then defined as \(\mathcal{B}=\{b_{1,1},b_{1,2},...,b_{1,|w_{1}|},...,b_{n,1},b_{n,2},...,b_{n,|w _{n}|}\}\), where \(|w_{i}|\) is the length of word \(w_{i}\), and \(b_{i,j}\) is the bridge-word obtained by first normalizing as described earlier and then removing the character at position \(j\) from the word \(w_{i}\in\mathcal{V}\) (it is possible that \(\mathcal{V}\cap\mathcal{B}\
eq\emptyset\)). These new words are used in addition to the original words when predicting their context in the skipgram model training, as depicted in Figure 1. For example, in the phrase 'that's my best Fri'endd ever', the objective is not only to predict 'that's,'my', 'best' and 'ever' using the word 'Fri'endd', but also using the derived bridge-words 'riend', 'fiend', 'frend', 'frand', 'frand', 'fried' and 'frien'. This idea of removing one character at a time is similar to the one used in the tool SymSpell ([[https://github.com/wolfgarbe/SymSpell](https://github.com/wolfgarbe/SymSpell)]([https://github.com/wolfgarbe/SymSpell](https://github.com/wolfgarbe/SymSpell))) to speed up spell-checking, where it replaces the exhaustive approach of considering all possible edit operations (i.e., addition, removal, substitution and transposition). In our case, bridge-words are not interesting _per se_ but as _intermediaries_ between other words. We do not require that they coincide with real words with which they would establish a direct connection; in fact, we assume that these connections will be indirect most of the time. For instance, we do not consider the substitution operations that would construct 'tome' and 'tame' from 'time',  
Figure 1: Visualization of the adapted skipgram model where we add bridge-words into its training. which would explicitly connect the three, but only 'tme', which can be obtained from the three of them by removing one character, linking them together indirectly. It is important to observe that these bridge-words also constitute artificial noise introduced in our training process that could play a harmful role. As an example, the word 'friend' appears as a bridge-word for 'friend', while also being a standard word from the English dictionary without much semantic relation to the concept of friendship.