Building a Text Normalizer using NLTK ft. POS tagger | by Nabanita Roy | Towards Data Science

Tokenization and text normalization are the two most fundamental steps of Natural Language Processing (NLP) techniques. Text normalization is the technique which transforms a word into its root or basic form in order to standardize text representation. Note that the basic form is not necessarily a meaningful word and we will see that by the end of this blog. These basic forms are termed as stems and the process is called stemming. A specialised approach to derive the stem of a word is called lemmatization which uses rules according to the part-of-speech(POS) family the word belongs to.
Text normalization is essential for Information Retrieval (IR) systems, data or text mining applications and NLP pipelines. To facilitate faster and efficient IR systems, indexing and searching algorithms require different word forms — derivational or inflectional — reduced to their normalized forms. Besides, it also helps in saving disk space and interlingual text matching.
The process of deriving lemmas deals with the semantics, morphology and the parts-of-speech(POS) the word belongs to, while Stemming refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. [1] This is the reason, in most applications, lemmatization has better performance over stemming, although narrowing down to a decision requires the bigger picture.
In Python’s NLTK Library, the nltk.stem package has both — stemmers and lemmatizers implemented.
Enough! Let’s dig into some text normalizing code now...
What could possibly go wrong??!
Note: Here is the complete jupyter notebook on GitHub.
Step 1: Tokenization
import nltkimport stringhermione_said = '''Books! And cleverness! There are more important things - friendship and bravery and - oh Harry - be careful!'''## Tokenizationfrom nltk import sent_tokenize, word_tokenizesequences = sent_tokenize(hermione_said)seq_tokens = [word_tokenize(seq) for seq in sequences]## Remove punctuationno_punct_seq_tokens = []for seq_token in seq_tokens:    no_punct_seq_tokens.append([token for token in seq_token if token not in string.punctuation])print(no_punct_seq_tokens)Output:[['Books'], ['And', 'cleverness'], ['There',  'are',  'more',  'important',  'things',  'friendship',  'and',  'bravery',  'and',  'oh',  'Harry',  'be',  'careful']]
Sweet!
Step 2: Normalization Techniques
We start with Stemming:
# Using Porter Stemmer implementation in nltkfrom nltk.stem import PorterStemmerstemmer = PorterStemmer()stemmed_tokens = [stemmer.stem(token) for seq in no_punct_seq_tokens for token in seq]print(stemmed_tokens)Output:['book', 'and', 'clever', 'there', 'are', 'more', 'import', 'thing', 'friendship', 'and', 'braveri', 'and', 'oh', 'harri', 'be', 'care']
So we have our first problem — out of the 16 tokens, the highlighted 3 does not look good!
All three are products of the crude heuristic process but ‘Harry’ to ‘harri’ is misleading, especially, for NER applications and ‘important’ to ‘import’ is information lost. ‘bravery’ to ‘bravery’ is an example of a stem that does not bear any meaning but could still be used in IR systems for indexing. A better example would be — argue,arguing, argued — all of which becomes ‘argu’, preserving the meaning of the original words but itself does not bear any meaning in the English dictionary.
Now, let’s lemmatize:
from nltk.stem.wordnet import WordNetLemmatizerfrom nltk.corpus import wordnetlm = WordNetLemmatizer()lemmatized_tokens = [lm.lemmatize(token) for seq in no_punct_seq_tokens for token in seq]print(lemmatized_tokens)Output:['Books', 'And', 'cleverness', 'There', 'are', 'more', 'important', 'thing', 'friendship', 'and', 'bravery', 'and', 'oh', 'Harry', 'be', 'careful']
Here, the previous three words that were incorrectly stemmed, look better. But, “Books” should have become “Book” just like “things” to “thing”.
That’s our second problem.
Now, WordNetLemmatizer.lemma() takes in an argument pos to understand the POS tag of the word/token because word-forms could be same but contextually or semantically different. For example:
print(lm.lemmatize("Books", pos="n"))Output: 'Books'print(lm.lemmatize("books", pos="v"))Output: 'book'
Therefore, taking help from POS tagger seems like a convenient option and we shall proceed to use POS tagger to solve this problem.
But before that, let’s look at which one to use — stemmer? lemmatizer? both? To answer this question, we need to take a step back and answer questions like:
What is the problem statement?
What features are important to address the problem statement?
Is this feature an overhead for computation and infrastructure?
The example above is a simple taster for the larger challenges that NLP practitioners face while processing millions of tokens’ basic forms. Besides, maintaining precision while processing huge corpora with additional checks like POS tagger (in this case), NER tagger, matching tokens in a Bag-of-Words(BOW) and spelling corrections are computationally expensive.
Step 3: POS Tagger to rescue
Let’s apply POS tagger on the already stemmed and lemmatized token to check their behaviours.
from nltk.tag import StanfordPOSTaggerst = StanfordPOSTagger(path_to_model, path_to_jar, encoding=’utf8')## Tagging Lemmatized Tokenstext_tags_lemmatized_tokens = st.tag(lemmatized_tokens)print(text_tags_lemmatized_tokens)Output:[('Books', 'NNS'), ('And', 'CC'), ('cleverness', 'NN'), ('There', 'EX'), ('are', 'VBP'), ('more', 'RBR'), ('important', 'JJ'), ('thing', 'NN'), ('friendship', 'NN'), ('and', 'CC'), ('bravery', 'NN'), ('and', 'CC'), ('oh', 'UH'), ('Harry', 'NNP'), ('be', 'VB'), ('careful', 'JJ')]## Tagging Stemmed Tokenstext_tags_stemmed_tokens = st.tag(stemmed_tokens)print(text_tags_stemmed_tokens)Output:[('book', 'NN'), ('and', 'CC'), ('clever', 'JJ'), ('there', 'EX'), ('are', 'VBP'), ('more', 'JJR'), ('import', 'NN'), ('thing', 'NN'), ('friendship', 'NN'), ('and', 'CC'), ('braveri', 'NN'), ('and', 'CC'), ('oh', 'UH'), ('harri', 'NNS'), ('be', 'VB'), ('care', 'NN')]
Spot the highlighted differences?
The first token — book — the stemmed token’s tag is NN (Noun) and that of the lemmatized one is NNS (Plural Noun), which more specific.
Second, Harry — wrongly stemmed to harri and as a result the POS tagger fails to identify it correctly as a Proper Noun. On the other hand, the lemmatized token correctly classified Harry and the POS tagger specifically identified it as a Proper Noun.
Lastly, harri and braveri — even though these words are not anywhere in the english lexical dictionary, they should have been classified as a FW. This I have no explanation or right now.
Step 4: Building the POS mapper for token tags
I first created a POS mapper like below. This mapper is for the arguments to wordnet according to the treebank POS tag codes.
from nltk.corpus.reader.wordnet import VERB, NOUN, ADJ, ADVdict_pos_map = {    # Look for NN in the POS tag because all nouns begin with NN    'NN': NOUN,    # Look for VB in the POS tag because all nouns begin with VB    'VB':VERB,    # Look for JJ in the POS tag because all nouns begin with JJ    'JJ' : ADJ,    # Look for RB in the POS tag because all nouns begin with RB    'RB':ADV  }
Step 5: Building the normalizer while addressing the problems
Identify the Proper Nouns and skips processing and retain Upper CaseIdentify the POS family the token’s POS tag belongs to — NN, VB, JJ, RB and pass the correct argument for lemmatizationGet the stems of the lemmatized tokens.
Identify the Proper Nouns and skips processing and retain Upper Case
Identify the POS family the token’s POS tag belongs to — NN, VB, JJ, RB and pass the correct argument for lemmatization
Get the stems of the lemmatized tokens.
Here, is the final code. I used st.tag_sents() to retain the order of the sequences (sentence-wise nested tokens)
With Stemming
normalized_sequence = []for each_seq in st.tag_sents(sentences=no_punct_seq_tokens):    normalized_tokens = []    for tuples in each_seq:        temp = tuples[0]        if tuples[1] == "NNP" or tuples[1] == "NNPS":            continue        if tuples[1][:2] in dict_pos_map.keys():            temp = lm.lemmatize(tuples[0].lower(),                                 pos=dict_pos_map[tuples[1][:2]])        temp = stemmer.stem(temp)        normalized_tokens.append(temp)    normalized_sequence.append(normalized_tokens)normalized_sequenceOutput:[['book'], ['and', 'clever'], ['there',  'be',  'more',  'import',  'thing',  'friendship',  'and',  'braveri',  'and',  'oh',  'be',  'care']]
Without Stemming
normalized_sequence = []for each_seq in st.tag_sents(sentences=no_punct_seq_tokens):    normalized_tokens = []    for tuples in each_seq:        temp = tuples[0]        if tuples[1] == "NNP" or tuples[1] == "NNPS":            continue        if tuples[1][:2] in dict_pos_map.keys():            temp = lm.lemmatize(tuples[0].lower(),                                 pos=dict_pos_map[tuples[1][:2]])        normalized_tokens.append(temp)    normalized_sequence.append(normalized_tokens)normalized_sequenceOutput:[['book'], ['And', 'cleverness'], ['There',  'be',  'more',  'important',  'thing',  'friendship',  'and',  'bravery',  'and',  'oh',  'be',  'careful']]
The two differences I still have in the process is important is stemmed to import and bravery to braveri. While “braveri” is considerably unambiguous, “import” poses a challenge because it could mean different things. This is why, in general, stemming has poorer performance than lemmatization as it increases search term ambiguities and therefore detects similarities which are not actually similar.
Note: Here is the complete jupyter notebook on GitHub.