## 5 NLP Operators for Enhancing Partial M2M Transformations
### NLP Techniques for Implementing Advanced Text Processing Operators  
To implement more advanced text processing operators (see Section 5.3), we utilized a set of well-established NLP techniques:  
Figure 6: Sample transformation pattern with SPLIT.  
* Lexical normalization deals with obtaining the initial word form. Two forms of normalization, namely, stemming and lemmatization, are widely known and applied in text processing. While stemming simply aims to reduce words to their base or root form, lemmatization depends on the part of speech and context and seeks to obtain the base form that is used in a dictionary. Besides dictionary-based lookup, statistical classifiers are used to get lemmas [66; 67].
* Tokenization splits the text into separate chunks (tokens). Simple tokenizers generally use predefined separator symbols, like white spaces or commas, to identify limits of such tokens. Real-world texts, however, may contain such characters inside the tokens themselves (abbreviations are one example). This is also different for Asian languages, which may require deeper morphological analysis. Modern toolkits include advanced tokenizers that can handle such problems or deal with specific situations, such as REPP tokenizer [68].
* Part-of-speech (POS) tagging identifies part-of-speech tags for each token in the sentence. This is one of the most researched topics in the natural language processing domain. Most POS tagging solutions usually rely on multiple statistical and machine learning approaches, such as convolutional networks (CNNs) [69], conditional random fields (CRFs) [70; 71], or transformation-based learning [72]. This is one of the key techniques used in noun phrase and verb phrase extraction, which is required to process element designations for proper generation of target elements.
* Named entity recognition (_NER_) focuses on finding entity instances in unstructured text and classifying them into a predefined set of categories, such as a person, organization, location, time, or address. Multiple approaches are available to solve this, including recent advances in deep learning [73; 74]. Still, hybrid conditional random field-based approaches seem to dominate in this field [75; 76; 77].
* Semantic analysis focuses on capturing synonyms, homonyms, antonyms, and other semantic relations. Lexical databases such as WordNet [62] can be directly applied to solve this task. Finding synonymous forms is one of the most popular tasks in the context of knowledge extraction, and is greatly beneficial for query processing or semantic entity-based searching. Recent developments in deep learning provide advanced techniques to find synonymous or contextually related entries by learning and comparing contextual representations in the form of redistributable embeddings [75; 78; 79]. These representations can be transferred and reused for different NLP tasks, including POS tagging, document classification, sentiment analysis, and others.
* Hypernym/hyponym discovery enables the extraction of hierarchical relationships to form taxonomies or augment existing ontologies or vocabularies. Rule-based [80; 81; 82], vector space [83], neural [84], and hybrid [85] approaches are among the most prominent ones in this category. In the context of our research, semantic analysis can be used to identify and merge synonymous forms and generate generalization relationships or categorizations.