Introduction to NLP - Part 1: Preprocessing text in Python | by Zolzaya Luvsandorj | Towards Data Science

Welcome to Introduction to NLP! This is the first part of the 5-part series of posts. This post will show one way to preprocess text using an approach called bag-of-words where each text is represented by its words regardless of the order in which they are presented or the embedded grammar. We will complete the following steps when preprocessing:
TokeniseNormaliseRemove stop wordsCount vectoriseTransform to tf-idf representation
Tokenise
Normalise
Remove stop words
Count vectorise
Transform to tf-idf representation
💤 Do these terms look like gibberish to you? Don’t worry, they will no longer be by the time you finish reading this post! 🎓
I assume the reader (👀 yes, you!) has access to and is familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have used and tested the scripts in Python 3.7.1. Let’s make sure you have the right tools before we get started.
We will use the following powerful third party packages:
pandas: Data analysis library,
nltk: Natural Language Tool Kit library and
sklearn: Machine Learning library.
The script below can help you download these corpora. If you have already downloaded, running this will notify you that they are up-to-date:
import nltknltk.download('stopwords') nltk.download('wordnet')
To keep things manageable, we will use tiny text data which will allow us to monitor inputs and outputs for each step. For this data, I have chosen Joey’s draft speech for Chandler and Monica’s wedding from the sitcom Friends.
His speech goes:
part1 = """We are gathered here today on this joyous occasion to celebrate the special love that Monica and Chandler share. It is a love based on giving and receiving as well as having and sharing. And the love that they give and have is shared and received. Andthrough this having and giving and sharing and receiving, we too can share and love and have... and receive."""part2 = """When I think of the love these two givers and receivers share I cannot help but envy the lifetime ahead of having and loving and giving and receiving."""
If you haven’t seen the part I am referring, there are short videos available on YouTube (keywords: Joey’s wedding speech). I think Joey’s acting and Monica and Chandler’s reaction definitely makes this speech much funnier than the mere text. Writing this post gave me a nice excuse to watch the scene repeatedly and I can’t have enough of it. 🙈
Firstly, let’s prepare the environment with packages and data:
# Import packages and modulesimport pandas as pdfrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import RegexpTokenizerfrom nltk.corpus import stopwordsfrom sklearn.feature_extraction.text import TfidfVectorizer# Create a dataframeX_train = pd.DataFrame([part1, part2], columns=['speech'])
Secondly, let’s define a text preprocessing function to pass it on to TfidfVectorizer:
def preprocess_text(text):    # Tokenise words while ignoring punctuation    tokeniser = RegexpTokenizer(r'\w+')    tokens = tokeniser.tokenize(text)        # Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    lemmas = [lemmatiser.lemmatize(token.lower(), pos='v') for token in tokens]        # Remove stopwords    keywords= [lemma for lemma in lemmas if lemma not in stopwords.words('english')]    return keywords
Lastly, let’s preprocess the text data by leveraging the function defined earlier:
# Create an instance of TfidfVectorizervectoriser = TfidfVectorizer(analyzer=preprocess_text)# Fit to the data and transform to feature matrixX_train = vectoriser.fit_transform(X_train['speech'])# Convert sparse matrix to dataframeX_train = pd.DataFrame.sparse.from_spmatrix(X_train)# Save mapping on which index refers to which wordscol_map = {v:k for k, v in vectoriser.vocabulary_.items()}# Rename each column using the mappingfor col in X_train.columns:    X_train.rename(columns={col: col_map[col]}, inplace=True)X_train
Ta-da❕ We have preprocessed text into feature matrix. Do these scripts make much sense without any explanation? Let’s break this down and understand the 5 steps mentioned at the beginning with examples in the next section.
💡 “Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation.”
In this step, we will convert a string part1 into list of tokens while discarding punctuation. There are many ways we could accomplish this task. I will show you one way to do so by using RegexpTokenizer from nltk:
# Import modulefrom nltk.tokenize import RegexpTokenizer# Create an instance of RegexpTokenizer for alphanumeric tokenstokeniser = RegexpTokenizer(r'\w+')# Tokenise 'part1' stringtokens = tokeniser.tokenize(part1)print(tokens)
Let’s inspect how tokens look like:
We see that each word is now a separate string. Do you notice how there are variations of the same word? For instance: words can differ in terms of their case: ‘and’ and ‘And’ or their suffix: ‘share’, ‘shared’ and ‘sharing’. This is where normalisation comes in to standardise.
💡 To normalise a word is to transform it into its root form.
Stemming and lemmatisation are popular ways to normalise text. In this step, we will use lemmatisation to transform words to their dictionary form as well as remove case distinction by converting all words to lowercase.
🔗 If you want to learn more about stemming and lemmatisation, you may want to check out the second part of the series.
We will use WordNetLemmatizer from nltk to lemmatise our tokens:
# Import modulefrom nltk.stem import WordNetLemmatizer# Create an instance of WordNetLemmatizerlemmatiser = WordNetLemmatizer()# Lowercase and lemmatise tokenslemmas = [lemmatiser.lemmatize(token.lower(), pos='v') for token in tokens]print(lemmas)
The words are now transformed to its dictionary form. For instance, ‘share’, ‘sharing’ and ‘shared’ are now all just ‘share’.
# Check how many words we havelen(lemmas)
We have 66 words but not all words carry the same level of contribution to the meaning of the text. In other words, there are some words that are not particularly useful to the key message. This is where stop words come in.
💡 Stop words are common words which provide little to no value to the meaning of the text.
Think about this: If you had to describe yourself in three words as elaborately as possible, would you include ‘I’, or ‘am’? If I asked you to underline keywords in Joey’s speech, would you underline ‘a’ or ‘the’? Probably not. The ‘I’, ‘am’, ‘a’ and ‘the’ are examples of stop words. I think you get the idea.
Different sets of stop words may be necessary depending on the domain that the text is related to. In this step, we will leverage nltk’s stopwords corpus. You could define your own set of stop words or enrich standard stop words by adding common terms that are appropriate to the domain of the text.
Let’s first familiarise ourselves with stopwords little bit more:
# Import modulefrom nltk.corpus import stopwords# Check out how many stop words there are print(len(stopwords.words('english')))# See first 5 stop wordsstopwords.words('english')[:5]
At the time of writing this post, there are 179 english stop words in the nltk’s stopword corpus. Some examples include: ‘i’, ‘me’, ‘my’, ‘myself’, ‘we’. If you are curious to see the full list, simply remove [:5] from the last line of code.
Notice how these stop words are in lowercase? To effectively remove stop words, we have to ensure that all words are in lowercase. Here, we have already done so in step two.
Using a list comprehension, let’s remove all stop words from our list:
keywords = [lemma for lemma in lemmas if lemma not in stopwords.words('english')]print(keywords)
# Check how many words we havelen(keywords)
After removing stop words, we only have 26 words as opposed to 66 yet the gist is still preserved.
Now, if you scroll back up to section 2 (Final Code) and have a quick look at the preprocess_text function, you will see that this function captures the transformation process shown in steps 1 to 3.
💡 Count vectorise is to convert a collection of text documents to a matrix of token counts.
Now let’s look at counts of each word in keywords from step 3:
{word: keywords.count(word) for word in set(keywords)}
The word ‘give’ occurs 3 times whereas ‘joyous’ was mentioned once.
This is essentially what CountVectorizer does to all records. CountVectorizer transforms text into a matrix of m by n where m is the number of text records, n is the number of unique tokens across all records and the elements of the matrix refer to the tally of a token for a given record.
In this step, we will convert our text dataframe to count matrix. We will pass our custom preprocessor function to CountVectorizer:
# Import modulefrom sklearn.feature_extraction.text import CountVectorizer# Create an instance of CountfVectorizervectoriser = CountVectorizer(analyzer=preprocess_text)# Fit to the data and transform to feature matrixX_train = vectoriser.fit_transform(X_train['speech'])
The output feature matrix will be in sparse_matrix form. Let’s convert it to a dataframe with proper column names to make it more readible:
# Convert sparse matrix to dataframeX_train = pd.DataFrame.sparse.from_spmatrix(X_train)# Save mapping on which index refers to which termscol_map = {v:k for k, v in vectoriser.vocabulary_.items()}# Rename each column using the mappingfor col in X_train.columns:    X_train.rename(columns={col: col_map[col]}, inplace=True)X_train
Once we transform it to dataframe, the columns would be just indices (i.e. numbers from 0 to n-1) instead of the actual words. Therefore, we need to rename the columns to make it easier to interpret.
When the vectoriser is fit to the data, we can find out the index mapping to words from vectoriser.vocabulary_. This index mapping is formatted as {word:index}. To rename columns, we must switch the key-value pairs to {index:word}. This is done in the second line of code and saved in col_map.
Using for loop at the end of the code, we are renaming each column using the mapping, and the output should look like what is in the table above (showing only partial output due to space limitation).
From this matrix, we can see ‘give’ has been mentioned 3 times in part1 (row index=0) and once in part2 (row index=1).
In our example, we only have 2 records each consisting of only a handful of sentences, so the count matrix is pretty small and its sparsity is not as high. Sparsity refers to the proportion of zero elements among all elements in a matrix. When you are working on real data with hundreds, thousands or even millions of records each represented by rich text, the count matrix is likely to be extremely large and contain mostly 0s. In those instances, using sparse format saves storage memory and speeds up further processing. As a result, you may not always convert sparse matrix to a dataframe like we did here for illustration when preprocessing text in real life.
💡 tf-idf stands for term frequency inverse document frequency.
When transforming to tf-idf representation, we are transforming the counts to weighted frequency where we give more significance to less frequent words and less importance to more frequent words by using a weight called inverse document frequency.
🔗 I have dedicated a separate post to explain this in detail in the third part of the series because I think it deserves its own section.
# Import modulefrom sklearn.feature_extraction.text import TfidfTransformer# Create an instance of TfidfTransformertransformer = TfidfTransformer()# Fit to the data and transform to tf-idfX_train = pd.DataFrame(transformer.fit_transform(X_train).toarray(), columns=X_train.columns)X_train
In the last step, we ensured that output is still a properly named dataframe:
Now that we understand step 4 and step 5 individually, I would like to point out that there is a more efficient way to complete steps 4 and 5 using TfidfVectorizer. Such is accomplished using the code below:
# Import modulefrom sklearn.feature_extraction.text import TfidfVectorizer# Create an instance of TfidfVectorizervectoriser = TfidfVectorizer(analyzer=preprocess_text)# Fit to the data and transform to tf-idfX_train = vectoriser.fit_transform(X_train['speech'])
To make this sparse matrix output into a dataframe with relevant column names, you know what to do (hint: see what we did in step 4).
Having covered all steps, if you go back to the scripts in section 2 (Final Code) again, does it look more familiar than the first time you saw? 👀
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for taking the time to go through this post. I hope that you learned something from reading it. Links to the rest of the posts are collated below:◼️ Part 1: Preprocessing text in Python◼️ Part 2: Difference between lemmatisation and stemming◼️ Part 3: TF-IDF explained◼️ Part 4: Supervised text classification model in Python◼️ Part 5A: Unsupervised topic model in Python (sklearn)◼️ Part 5B: Unsupervised topic model in Python (gensim)
Happy preprocessing! Bye for now 🏃💨
Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press, 2008
Bird, Steven, Edward Loper and Ewan Klein, Natural Language Processing with Python. O’Reilly Media Inc, 2009
Feature Extraction, sklearn documentation