Text Processing in Python. Text processing example with NLTK and... | by KahEm Chu | Towards Data Science

The Internet has connected the world, while Social Media like Facebook, Twitter and Reddit provided the platform for people to express their opinions and feelings toward a topic. Then, the proliferation of smartphones increased the usage of these platforms directly. For instance, there are 96% or 2,240 million Facebook active users who used Facebook by smartphones and tablets [1].
The increment in the usage of Social Media has grown the size of text data, and boost the studies or researches in Natural Language Processing (NLP), for example, Information Retrieval and Sentiment Analysis. Most of the time, the documents or the text files to be analyzed are gigantic and contains a lot of noise, directly used raw texts for analysis is inapplicable. Hence, text processing is essential to provide clean input for modelling and analysis.
Text processing contains two main phases, which are tokenization and normalization [2]. Tokenization is the process of splitting a longer string of text into smaller pieces, or tokens [3]. Normalization referring to convert number to their word equivalent, remove punctuation, convert all text to the same case, remove stopwords, remove noise, lemmatizing and stemming.
Stemming — removing affixes (suffixed, prefixes, infixes, circumfixes), For example, running to run
Lemmatization — capture canonical form based on a word’s lemma. For example, better to good [4]
In this article, I am going to demonstrate text processing in Python.
The following is a paragraph extracted from the book “Speech and Language Processing” by Daniel Jurafsky and James H. Martin [6].
“The idea of giving computers the ability to process human language is as old as the idea of computers themselves. This book is about the implementation and implications of that exciting idea. We introduce a vibrant interdisciplinary field with many names corresponding to its many facets, names like speech and language processing, human language technology, natural language processing, computational linguistics, and speech recognition and synthesis. The goal of this new field is to get computers to perform useful tasks involving human language, tasks like enabling human-machine communication, improving human-human communication, or simply doing useful processing of text or speech.”
This paragraph will be used in the examples of text processing below.
For text processing in Python, two Natural Language Processing (NLP) libraries, namely NLTK (Natural Language Toolkit) and spaCy will be used in the demonstration. The rationale behind selecting these two libraries instead of other text processing libraries like Gensim and Transformer is because NLTK and spaCy are the most popular library for everyone and friendly for the beginner in Natural Language Processing (NLP).
For both NLTK and spaCy, the text will need to be saved as a variable in the first place.
text = """The idea of giving computers the ability to process human language is as old as the idea of computers themselves. This book is about the implementation and implications of that exciting idea. We introduce a vibrant interdisciplinary field with many names corresponding to its many facets, names like speech and language processing, human language technology, natural language processing, computational linguistics, and speech recognition and synthesis. The goal of this new field is to get computers to perform useful tasks involving human language, tasks like enabling human-machine communication, improving human-human communication, or simply doing useful processing of text or speech."""
Import all needed libraries
Import all needed libraries
import reimport pandas as pdimport nltkfrom nltk.tokenize import WordPunctTokenizernltk.download(’stopwords’)from nltk.corpus import stopwords# needed for nltk.pos_tag function nltk.download(’averaged_perceptron_tagger’)nltk.download(’wordnet’)from nltk.stem import WordNetLemmatizer
Tokenization
Tokenization
Using tokenizer to separate the sentences into a list of single words (tokens).
word_punct_token = WordPunctTokenizer().tokenize(text)
There are several tokenizer modules in NLTK libraries other than WordPunctTokenizer used above. For example, word_tokenizeand RegexpTokenizer. RegexpTokenizer able to separate the currency like $9.99 as a single token by setting RegexpTokenizer(‘\w+|\$[\d\.]+|\S+’). All the tokenizers mentioned will be return the tokens in the list form. NLTK also have a module name sent_tokenize which able to separate paragraphs into the list of sentences.
2. Normalization
The script below removed the tokens which are not a word, for example, the symbols and numbers, also tokens that only contain less than two letters or contain only consonants. This script might not be useful in this example, but it’s pretty useful when you are dealing with a large set of text data, it helps to clean up much noise. I always like to include it whenever I was processing text data.
clean_token=[]for token in word_punct_token:    token = token.lower()    # remove any value that are not alphabetical    new_token = re.sub(r'[^a-zA-Z]+', '', token)     # remove empty value and single character value    if new_token != "" and len(new_token) >= 2:         vowels=len([v for v in new_token if v in "aeiou"])        if vowels != 0: # remove line that only contains consonants            clean_token.append(new_token)
2a. Removing Stopwords
Stopwords referring to the word which does not carry much insight, such as preposition. NLTK and spaCy have a different amount of stopwords in the library, but both NLTK and spaCy allowed us to add in any word we feel necessary. For example, when we dealing with email, we may add in gmail, com, outlook as stopwords.
# Get the list of stop wordsstop_words = stopwords.words('english')# add new stopwords to the liststop_words.extend(["could","though","would","also","many",'much'])print(stop_words)# Remove the stopwords from the list of tokenstokens = [x for x in clean_token if x not in stop_words]
2b. Part-of-Speech Tagging (POS Tag)
This process referring to tag the word with their part-of-speech position, for example, verbs, adjectives and nouns. nltk.pos_tag module return results as tuples, to ease the task afterwards, normally I will transform them into a DataFrame. In [6], the POS Tag is the task perform straight after tokenization, which is a smart move when you know you only need a particular part of speech like adjectives and nouns.
data_tagset = nltk.pos_tag(tokens)df_tagset = pd.DataFrame(data_tagset, columns=['Word', 'Tag'])
2c. Lemmatization
Lemmatizing and stemming both help to reduce the dimension of the vocabulary by return the words to their root form (lemmatizing) or remove all the suffix, affix, prefix and so on (stemming). Stemming is nice for reducing the dimension of vocabulary, but most of the time the word become meaningless as stemming only chopped off the suffix but not returning the words to their base form. For example, houses will become hous after stemming, which completely lose its meaning. Hence, lemmatizing is more preferable for text analytics.
The following script is used to obtain the root form for the nouns, adjectives and verbs.
# Create lemmatizer object lemmatizer = WordNetLemmatizer()# Lemmatize each word and display the outputlemmatize_text = []for word in tokens:    output = [word, lemmatizer.lemmatize(word, pos='n'),       lemmatizer.lemmatize(word, pos='a'),lemmatizer.lemmatize(word, pos='v')]    lemmatize_text.append(output)# create DataFrame using original words and their lemma wordsdf = pd.DataFrame(lemmatize_text, columns =['Word', 'Lemmatized Noun', 'Lemmatized Adjective', 'Lemmatized Verb']) df['Tag'] = df_tagset['Tag']
The reason for lemmatizing the adjectives, nouns and verbs separately is to improve the accuracy of the lemmatizer.
# replace with single character for simplifyingdf = df.replace(['NN','NNS','NNP','NNPS'],'n')df = df.replace(['JJ','JJR','JJS'],'a')df = df.replace(['VBG','VBP','VB','VBD','VBN','VBZ'],'v')'''define a function where take the lemmatized word when tagset is noun, and take lemmatized adjectives when tagset is adjective'''df_lemmatized = df.copy()df_lemmatized['Tempt Lemmatized Word']=df_lemmatized['Lemmatized Noun'] + ' | ' + df_lemmatized['Lemmatized Adjective']+ ' | ' + df_lemmatized['Lemmatized Verb']df_lemmatized.head(5)lemma_word = df_lemmatized['Tempt Lemmatized Word']tag = df_lemmatized['Tag']i = 0new_word = []while i<len(tag):    words = lemma_word[i].split('|')    if tag[i] == 'n':                word = words[0]    elif tag[i] == 'a':        word = words[1]    elif tag[i] == 'v':        word = words[2]    new_word.append(word)    i += 1df_lemmatized['Lemmatized Word']=new_word
The script above is to assign the correct lemmatized word to the original word according to their POS Tag.
3. Obtain the Cleaned Tokens
# calculate frequency distribution of the tokenslemma_word = [str(x) for x in df_lemmatized['Lemmatized Word']]
After tokenized and normalized the text, now we obtained a list of clean tokens, which ready to be plug-in into WordCloud or other text analytics models.
Tokenization + Lemmatization
Tokenization + Lemmatization
SpaCy advantage compared to NLTK is it simplified the text processes.
# Import spaCy and load the language libraryimport spacy#you will need this line below to download the package!python -m spacy download en_core_web_smnlp = spacy.load('en_core_web_sm')# Create a Doc objectdoc = nlp(text)token_list = []# collect each token separately with their POS Tag, dependencies and lemmafor token in doc:    output = [token.text, token.pos_, token.dep_,token.lemma_]    token_list.append(output)# create DataFrame using data df = pd.DataFrame(token_list, columns =['Word', 'POS Tag', 'Dependencies', 'Lemmatized Word']) 
In spaCy, you may obtain the POS Tag and the lemma (root form of the word) when you performing tokenization, which saves you some effort.
2. Normalization
As the lemmatizing is performed at the very beginning, the normalization steps leave only removing noise and stopwords.
2a. Removing Noise
df_nopunct = df[df['POS Tag']!='PUNCT']df_nopunct
The spaCy POS Tag contained a punctuation tag which showed as PUNCT, hence we can remove all the punctuation by removing the tokens with PUNCT tag.
2b. Removing Stopwords
import numpy as nplemma_word = df_nopunct['Lemmatized Word'].values.tolist()stopword = nlp.Defaults.stop_words# Add the word to the set of stop words. Use lowercase!nlp.Defaults.stop_words.add('btw')is_stopword_list = []for word in lemma_word:    is_stopword = nlp.vocab[word].is_stop    is_stopword_list.append(is_stopword)df_nopunct["is_stopword"] = is_stopword_listdf_nopunctclean_df = df_nopunct[df_nopunct["is_stopword"]==False]
SpaCy have a class .is_stop which can be used to detect whether a token is a stopword, we can use it to remove the stopwords as shown in the script above.
3. Obtain the Cleaned Tokens
clean_list = clean_df["Lemmatized Word"].values.tolist()
Now we obtained the list of cleaned tokens!
spaCy is faster and more advance than NLTK, but NLTK is better for beginners to understand every process in text processing as it occurs longer and does have a lot of documentation and explanation. You may find the comparison of spaCy and NLTK for other features like GPU support, efficiency, performance, state of the art, word vectors and flexibility here.
Thank you for reading until here! Hope the demonstration above does help you to understand text processing in Python.
For text processing, understanding the data structure is as important as Regular Expression (RegEx). For example, the return of the class or module might be in the form of tuples and lists, to manipulate the return output, we first have to understand them. The below is some of the scripts I used frequently to change the data structure of the variables for different purposes.
From DataFrame (Multiple Columns) to Nested List, From Nested List to List
From DataFrame (Multiple Columns) to Nested List, From Nested List to List
When we are creating Topic Modelling or Sentiment Analysis models, normally we kept the tokens separately according to sentences or paragraphs, for example, reviews and tweets are kept separately to obtain the accurate sentiment or topic for each review or tweet. Hence, if we have 10 reviews, there will be 10 lists of tokens, saving them together in a single variable create a nested list. Another example is, a DataFrame with few columns holding text data, which could be essay questions for survey or maybe reviews for a product. We might want to change these columns into lists directly, but changing few columns of data into a list created a nested list also.
# df_essays is a dataframe with few columns of text dataessay_list = df_essays.values.tolist() # this create a nested list# this create a flat listflatEssayList = [item for elem in essay_list for item in elem if str(item) != 'nan']
2. From DataFrame (Single Columns) or List to Text
One of the popular application in Text Analytics is WordCloud, which mainly used to analyze the most frequent discussion in the text. This module only takes text as input, so in this scenario, we will need to change the list or DataFrame.Column to text.
# dataframe.column to texttext = ‘ ‘.join(str(x) for x in df[‘review’])# list to texttext = ‘ ‘.join(str(x) for x in any_list)
3. From DataFrame (Single Column) to List
reviews_list = df["reviews"].values.tolist()
4. From Text to List
This is simple and handy.
token_list = text.split()
5. From Tuples to DataFrame
I mentioned this above, repeat it here for better reference.
The nltk.pos_tagmodule returns the tag set as tuples in form of (word, pos tag) .
data_tagset = nltk.pos_tag(tokens)df_tagset = pd.DataFrame(data_tagset, columns=['Word', 'Tag'])
If you are interested in extracting the sentences that contain the keywords from a keyword list: Text Extraction using Regular Expression (Python).
If you are interested in explore the gigantic text with Word Cloud: Text Exploration with Word Cloud.
Subscribe on YouTube
[1] M. Iqbal, “Facebook Revenue and Usage Statistics (2020),” 8 March 2021. [Online]. Available: https://www.businessofapps.com/data/facebook-statistics/.
[2] M. Mayo, “A General Approach to Preprocessing Text Data,” 2017. [Online]. Available: https://www.kdnuggets.com/2017/12/general-approach-preprocessing-text-data.html. [Accessed 12 June 2020].
[3] D. Subramanian, “Text Mining in Python: Steps and Examples,” 22 August 2019. [Online]. Available: https://medium.com/towards-artificial-intelligence/text-mining-in-python-steps-and-examples-78b3f8fd913b. [Accessed 12 June 2020].
[4] M. Mayo, “Natural Language Processing Key Terms, Explained,” 2017. [Online]. Available: https://www.kdnuggets.com/2017/02/natural-language-processing-key-terms-explained.html.
[5] “Natural Language Processing In Julia (Text Analysis),” JCharisTech, 1 May 2018. [Online]. Available: https://jcharistech.wordpress.com/2018/05/01/natural-language-processing-in-julia-text-analysis/.
[6] D. Jurafsky and J. H. Martin, “Speech and Language Processing,” 3 December 2020. [Online]. Available: https://web.stanford.edu/~jurafsky/slp3/.
[7] M.F. Goh, “Text Normalization with spaCy and NLTK,” 29 November 2020. [Online]. Available: https://towardsdatascience.com/text-normalization-with-spacy-and-nltk-1302ff430119.
Congrats and thanks for reading to the end. Hope you enjoy this article. ☺️