Abstract: We critically evaluate the widespread assumption that deep learning NLP models do not require lemmatized input. To test this, we trained versions of contextualised word embedding ELMo models on raw tokenized corpora and on the corpora with word tokens replaced by their lemmas. Then, these models were evaluated on the word sense disambiguation task. This was done for the English and Russian languages.  ::: The experiments showed that while lemmatization is indeed not necessary for English, the situation is different for Russian. It seems that for rich-morphology languages, using lemmatized training and testing data yields small but consistent improvements: at least for word sense disambiguation. This means that the decisions about text pre-processing before training ELMo should consider the linguistic nature of the language in question.

Introduction: Deep contextualised representations of linguistic entities (words and/or sentences) are used in many current state-of-the-art NLP systems. The most well-known examples of such models are arguably ELMo BIBREF0 and BERT BIBREF1. A long-standing tradition if the field of applying deep learning to NLP tasks can be summarised as follows: as minimal pre-processing as possible. It is widely believed that lemmatization or other text input normalisation is not necessary. Advanced neural architectures based on character input (CNNs, BPE, etc) are supposed to be able to learn how to handle spelling and morphology variations themselves, even for languages with rich morphology: `just add more layers!'. Contextualised embedding models follow this tradition: as a rule, they are trained on raw text collections, with minimal linguistic pre-processing. Below, we show that this is not entirely true. It is known that for the previous generation of word embedding models (`static' ones like word2vec BIBREF2, where a word always has the same representation regardless of the context in which it occurs), lemmatization of the training and testing data improves their performance. BIBREF3 showed that this is true at least for semantic similarity and analogy tasks. In this paper, we describe our experiments in finding out whether lemmatization helps modern contextualised embeddings (on the example of ELMo). We compare the performance of ELMo models trained on the same corpus before and after lemmatization. It is impossible to evaluate contextualised models on `static' tasks like lexical semantic similarity or word analogies. Because of this, we turned to word sense disambiguation in context (WSD) as an evaluation task. In brief, we use contextualised representations of ambiguous words from the top layer of an ELMo model to train word sense classifiers and find out whether using lemmas instead of tokens helps in this task (see Section SECREF5). We experiment with the English and Russian languages and show that they differ significantly in the influence of lemmatization on the WSD performance of ELMo models. Our findings and the contributions of this paper are: Linguistic text pre-processing still matters in some tasks, even for contemporary deep representation learning algorithms. For the Russian language, with its rich morphology, lemmatizing the training and testing data for ELMo representations yields small but consistent improvements in the WSD task. This is unlike English, where the differences are negligible.

Question: Why is lemmatization not necessary in English?
Answer: Advanced neural architectures and contextualized embedding models learn how to handle spelling and morphology variations.