Text Normalization with spaCy and NLTK | by Manfye Goh | Towards Data Science

Most NLP tasks require us to refer to a dictionary to teach the machine the word’s context or vocabulary, it is locally to think that, the smaller the vocabulary the better the performance of our NLP task.
Every NLP pipeline needs to do text normalization. Text normalization is the process of transforming a text into a canonical (standard) form. It is one of the important steps in text preprocessing to reduce the noises generated by a single word with multiple forms. For example: Connect, connected, connects all refer to the word “connect”, it is hence easier for us to search for 1 word in the dictionary than searching for 3 words.
2 important task in text normalization are:
Stemming - The process of reducing a word to its stem or root format.
Lemmatization - The transformation that uses a dictionary to map a word’s variant back to its root format
The real difference between stemming and lemmatization is that Stemming reduces word-forms to (pseudo)stems which might be meaningful or meaningless, whereas lemmatization reduces the word-forms to linguistically valid meaning.
Towards my learning of learning NLP in python, I noticed that the major obstacle is that there are too many libraries for text processing in python, to name a few: Gensim, TextBlob, spaCy, Pattern, Standford CoreNLP, and NLTK. These libraries do provide their own advantages over another but it creates confusion for the beginner.
This article will explore the text normalization process in the two libraries that widely use in the NLP domain which are NLTK and spaCy. Below is a brief comparison of both library:
Let's start with some random tweet I found on Twitter:
You can get the full code here
Example_Sentence = "Patients who in late middle age have smoked 20 cigarettes a day since their teens constitute an at-risk group. One thing they’re clearly at risk for is the acute sense of guilt that a clinician can incite, which immediately makes a consultation tense."#You might need to download the model when you first load the model, follow the instruction given in the error message.
#spaCy Code Initialization:import spacynlp = spacy.load('en_core_web_sm')doc = nlp(Example_Sentence)
nlp() will subject the sentence into the NLP pipeline of spaCy, and everything is automated as the figure above, from here, everything needed is tagged such as lemmatization, tokenization, NER, POS. What we going to do next is just extract the processed token.
#NLTK Code Initialization:import nltkfrom nltk.tokenize import word_tokenize  from nltk.corpus import stopwordsfrom nltk.stem import WordNetLemmatizerfrom nltk.stem.snowball import SnowballStemmerfrom nltk.stem.porter import *
The normal process of importing libraries. Here we imported tokenize library, stopwords list, lemmatizer, porter stemmer, snowball stemmer.
#spaCytoken_listSC = []for token in doc:    token_listSC.append(token.text)print(token_listSC)
Note: This step in spaCy is only needed when you just want the pure tokenized list.
#nltknltk_tokenList = word_tokenize(Example_Sentence)
For your information, spaCy doesn’t have a stemming library as they prefer lemmatization over stemmer while NLTK has both stemmer and lemmatizer
p_stemmer = PorterStemmer()nltk_stemedList = []    for word in nltk_tokenList:        nltk_stemedList.append(p_stemmer.stem(word))
The 2 frequently use stemmer are porter stemmer and snowball stemmer. The example above show porter stemmer
#spaCylemma_list = []for token in doc:    lemma_list.append(token.lemma_)print("Tokenize+Lemmatize:")print(lemma_list)
Lemmatization in spaCy is just extracting the processed doc from the spaCy NLP pipeline. As you can see here, one line of code is able to do tokenization and lemmatization together (So wonderful!)
#NLTKwordnet_lemmatizer = WordNetLemmatizer()    nltk_lemmaList = []    for word in nltk_stemedList:        nltk_lemmaList.append(wordnet_lemmatizer.lemmatize(word))
NLTK use wordnet lemmatizer, you have to import wordnet lemmatizer to do lemmatization.
Below is the code for the function, kindly cancel the process you do not need when using.
Tokenization → Lemmatization → Remove stopwords → Remove punctuation
def spacy_process(text):    doc = nlp(text)    #Tokenization and lemmatization are done with the spacy nlp pipeline commands    lemma_list = []    for token in doc:        lemma_list.append(token.lemma_)    print("Tokenize+Lemmatize:")    print(lemma_list)        #Filter the stopword    filtered_sentence =[]     for word in lemma_list:        lexeme = nlp.vocab[word]        if lexeme.is_stop == False:            filtered_sentence.append(word)         #Remove punctuation    punctuations="?:!.,;"    for word in filtered_sentence:        if word in punctuations:            filtered_sentence.remove(word)    print(" ")    print("Remove stopword & punctuation: ")    print(filtered_sentence)
Tokenization → Stemming → Lemmatization → Remove stopwords → Remove punctuation
import nltkfrom nltk.tokenize import word_tokenize  from nltk.corpus import stopwordsfrom nltk.stem import WordNetLemmatizerfrom nltk.stem.snowball import SnowballStemmerfrom nltk.stem.porter import *p_stemmer = PorterStemmer()def nltk_process(text):    #Tokenization    nltk_tokenList = word_tokenize(text)        #Stemming    nltk_stemedList = []    for word in nltk_tokenList:        nltk_stemedList.append(p_stemmer.stem(word))        #Lemmatization    wordnet_lemmatizer = WordNetLemmatizer()    nltk_lemmaList = []    for word in nltk_stemedList:        nltk_lemmaList.append(wordnet_lemmatizer.lemmatize(word))        print("Stemming + Lemmatization")    print(nltk_lemmaList)    #Filter stopword    filtered_sentence = []      nltk_stop_words = set(stopwords.words("english"))    for w in nltk_lemmaList:          if w not in nltk_stop_words:              filtered_sentence.append(w)    #Removing Punctuation    punctuations="?:!.,;"    for word in filtered_sentence:        if word in punctuations:            filtered_sentence.remove(word)    print(" ")    print("Remove stopword & Punctuation")    print(filtered_sentence)
*Note: It is normal to remove stopwords and punctuation first before tokenization. However, it doesn’t make any major difference in both libraries processing
Notice there are differences in the outcome, the result of NLTK tends to be more unread-able due to the stemming process while both libraries also reduce the token count to 27 tokens.
If you noticed in the spaCy result, spaCy adds a special case for English pronouns: all English pronouns are lemmatized to the special token -PRON-. Besides that, the word “tense” is preserved, while the stemming process in NLTK reduces the word to “ten” which might not represent the original meaning.
Regarding the processing time, spaCy use 15ms as compare to NLTK used 1.99ms in my example. It is reported that spaCy is way faster than NLTK, however, it is not shown here. Perhaps we need a larger example to show the superiority of spaCy in terms of speed.
You can get the full code here
While searching for the resources to write this article, I noticed that there are even more libraries that use to perform similar tasks as spaCy and NLTK. However, these libraries are less popular and lack maintenance.
I personally will learn both libraries as in my NLP journey with more focuses on spaCy as NLTK is quite an old library and unable to cope with newer machine learning models such as transformers while spaCy 3.0 will release the transformer-based pipelines and have the accuracy right up to the current state-of-the-art. The only drawback is that spaCy can’t process other languages except for English.
Lastly,
Thank you for reading, I appreciated your time for reading.