# Towards Robust Word Embeddings for Noisy Texts
## Appendix A The Importance of Word Segmentation  
In principle, when using word embeddings, we assume that the input text is correctly segmented into words. However, suppose that this is not the case, and that it goes beyond frequent de-normalization instances where words are joined or merged together; e.g., 'noway'-'no way', 'yesplease'-'yes please'; or even instances where the individual words cannot be immediately recovered such as with 'tryna'-'trying to', or 'whatchu'-'what are/do you'. Instead, in the present case, we will consider sentences like 'theproblem was veryclear' or 'the prob lem was very clear'. A possible solution would be to perform a word segmentation preprocessing step [50] before obtaining the corresponding word embeddings, which would imply introducing again the notion of sequential tasks together with the risk of error propagation.  
However, let us now consider that the two operations involved in bad word segmentation (i.e., word joining and splitting) might not have the same impact on the process of obtaining relevant word embeddings. If we take into account that models such as fastText, and by extension the modification presented in this chapter, use subword information to construct word embeddings, we might argue that _joining words together_ may be moderately supported by these models, as they would still consider the words inside the merging as character n-grams modelled during training. On the contrary, _splitting words_ would be more problematic, as it removes parts of a word which could be crucial to obtain the adequate vector representation.  
To check this hypothesis, we have devised new experiments using new de-normalized versions of the STS* datasets from the SentEval benchmark, which we have divided into two sets: _join_ and _split_. In the former, we randomly removed word delimiters from input sentences with a fixed probability \(p_{j}\), while in the latter we added delimiters between word characters with a lower probability \(p_{s}\), \(p_{s}<p_{j}\), in order to account for the higher amount of non-delimiter characters.  
The results obtained, which are shown in Table 1, seem to support our hypothesis. Therefore, using a word segmenter with a slight tendency to join words (e.g., through a threshold parameter as shown by [PERSON] et al. [51]) or even the raw input directly (taking into account the low frequency of splits, while joins are frequent in special elements such as hashtags or URLs), can be considered good practical solutions so long as we use embedding models that exploit subword information. Nonetheless, the latter option is especially relevant for us, since it shows that we may finally dispense with any form of input preprocessing for languages that delimit words; English in our current case. However, even in the case of Chinese, where words are not explicitly delimited and word segmentation is a well-studied and complex subject, it has been recently shown that this preprocessing step might not be necessary. [PERSON] et al. [52] propose directly operating over Chinese characters rather than _strict_ words. We highlight this strictness property as characters are frequently used as words themselves, but not always. That solution obtains better results than other systems that require a previous word segmentation step, even when all of them are implemented as state-of-the-art neural networks. In our case, this shows that we could relax the definition of a _word_, and obtain the embeddings at the character- or sequence-of-characters level.