Preprocessing text in Python. A step towards building a sentiment... | by Zolzaya Luvsandorj | Towards Data Science

This post is the second of three sequential posts on steps to build a sentiment classifier. Following our exploratory text analysis in the first post, it’s time to preprocess our text data. Simply put, preprocessing text data is to do a series of operations to convert the text into a tabular numeric data. In this post, we will look at 3 ways with varying complexity to preprocess text to tf-idf matrix as preparation for a model. If you are unsure what tf-idf is, this post explains with a simple example.
Before we dive in, let’s take a step back and look at the bigger picture real quickly. CRISP-DM methodology outlines the process flow for a successful data science project. Preprocessing data is one of the key tasks in the data preparation stage.
This post assumes that the reader (👀 yes, you!) has access to and is familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have tested the scripts in Python 3.7.1 in Jupyter Notebook.
Let’s make sure you have the following libraries installed before we start:◼️ Data manipulation/analysis: numpy, pandas◼️ Data partitioning: sklearn◼️ Text preprocessing/analysis: nltk◼️ Spelling checker: spellchecker (pyspellchecker when installing)
Once you have nltk installed, please make sure you have downloaded ‘stopwords’ and ‘wordnet’ corpora from nltk with the script below:
import nltknltk.download('stopwords') nltk.download('wordnet')
If you have already downloaded, running this will notify you so.
Now, we are ready to import all the packages:
# Setting random seedseed = 123# Measuring run timefrom time import time# Data manipulation/analysisimport numpy as npimport pandas as pd# Data partitioningfrom sklearn.model_selection import train_test_split# Text preprocessing/analysisimport re, randomfrom nltk import word_tokenize, sent_tokenize, pos_tagfrom nltk.util import ngramsfrom nltk.corpus import stopwordsfrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import RegexpTokenizerfrom sklearn.feature_extraction.text import TfidfVectorizerfrom spellchecker import SpellChecker
We will use IMDB movie reviews dataset. You can download the dataset here and save it in your working directory. Once saved, let’s import it to Python:
sample = pd.read_csv('IMDB Dataset.csv')print(f"{sample.shape[0]} rows and {sample.shape[1]} columns")sample.head()
Let’s look at the split between sentiments:
sample['sentiment'].value_counts()
Sentiment is evenly split in the sample data. Let’s first partition the data into two sets: train and test. We will set aside 5000 cases for testing:
# Split data into train & testX_train, X_test, y_train, y_test = train_test_split(sample['review'], sample['sentiment'], test_size=5000, random_state=seed,                                                     stratify=sample['sentiment'])# Append sentiment back using indicestrain = pd.concat([X_train, y_train], axis=1)test = pd.concat([X_test, y_test], axis=1)# Check dimensionsprint(f"Train: {train.shape[0]} rows and {train.shape[1]} columns")print(f"{train['sentiment'].value_counts()}\n")print(f"Test: {test.shape[0]} rows and {test.shape[1]} columns")print(test['sentiment'].value_counts())
We will be using train for the preprocessing experiments in this post. We will preprocess the test only when we need to assess the final model. Let’s inspect the head of the training dataset:
train.head()
Alright, let’s start preprocessing! ✨
Depending on how we process, we could arrive at different tf-idf matrices. When building a model, it’s good to try out different ways of preprocessing. We will look at the following 3 approaches:
Simpler approachSimple approachLess simple approach
Simpler approach
Simple approach
Less simple approach
Although we are only looking at preprocessing text to tf-idf matrix in this post, you may also want to explore other alternatives.
For each approach, we will be sure to measure run time performance since it is an important consideration especially if the model is going to be productionised. We will test which approach works better with the model in the next post. Before we dive in, to make things easier, let’s define a function that help us inspect the different approaches:
def inspect(vectoriser, X):    # Fit and transform    start = time()    print(f"There are {vectoriser.fit_transform(X).shape[1]} columns.\n")    end = time()    print(f"Took {round((end-start),2)} seconds.\n")        # Inspect tokens    tokens = list(vectoriser.vocabulary_.keys())    tokens.sort()    print(f"Example tokens: {tokens[:50]}\n")        # Inspect ignored tokens    ignored = vectoriser.stop_words_    if len(ignored)==0:        print("No token is ignored.")    elif len(ignored)>50:        print(f"Example ignored tokens: {random.sample(ignored, 50)}")    else:        print(f"Example ignored tokens: {ignored}")
Let’s start with the simplest of the three. In this approach, we will let sklearn’s TfidfVectorizer do all the preprocessing without using any additional custom functions. Let’s use the default parameters to get a feel of how many columns we will get:
vectoriser = TfidfVectorizer()inspect(vectoriser, X_train)
It was pretty quick. The output has over 97 thousand columns and the first 50 tokens are mostly numbers. Now, let’s tweak a few of the parameters to clean up the tokens further and better control the preprocessing:
# Simpler approachvectoriser = TfidfVectorizer(token_pattern=r'[a-z]+', stop_words='english', min_df=30, max_df=.7)inspect(vectoriser, X_train)
While the two versions above take about the same time, the number of columns are 7–8 times lower in the latter. Here, we have asked TfidfVectorizer to do a few things:◼ token_pattern=r'[a-z]+': tokenise into alphabetic tokens - this means we discard numbers and punctuation. If you are not familiar with regular expression, [a-z]+ means a token must consist of only letters. ◼ stop_words='english’: remove stop words.◼ min_df=30: remove rare tokens. We are considering a token as rare when it’s in less than 30 reviews. This will shrink the number of tokens significantly. Try running the script without this parameter and see the number of tokens.◼ max_df=.7: remove tokens that are in more than 70% of the documents. This means if a token is contained in more than 31.5K reviews, then we are ignoring them. There aren’t actually many words excluded because of this. So, we could even keep this feature to its default.
max_df and min_df help with token selection. In other words, with the help of these two, we can drop tokens that are either too frequent that may not be useful in classifying sentiments or too rare that may contribute to overfitting.
📍 Here’s a tip: If we define the parameters like this:
TfidfVectorizer(token_pattern=r'[a-z]+', max_df=.5)
you can find out what tokens were excluded because of max_df=.5 condition by running vectoriser.stop_words_ after fitting the vectoriser. When we call inspect function, the ‘example ignored tokens’ part of the output shows a snippet of those excluded words. Now, I encourage you to run the script above and check vectoriser.stop_words_. What do you see? Do you see mostly stop words? Try changing the values to .5, .6, .8 or .9 and rerun to observe how the excluded words change. It helps to understand how tweaking certain parameters impacts the preprocessing. Now, if you are keen, add stop_words='english’ or min_df=30 (don’t add both at the same time, add one at a time to understand individual parameters) and check out what tokens were excluded this time. We will tune these parameters further when building a model in the next post.
🔗 If you are keen to learn more about parameters , the documentation is here.
This approach was a 🍰, wasn’t it? I like this approach especially because it does a few basic things really well and quick. In my opinion, it’s always good to start simple and add complexity only if it improves performance.
There is always room for improvement. For instance, the word ‘play’, ‘plays’, ‘playing’ and ‘played’ are considered as 4 different tokens in previous approach. Wouldn’t it be great if we remove these inflectional endings so that these tokens are normalised into 1 unique token, ‘play’? That is what we are going to do in the this section!
To normalise inflectional endings, we will use a technique called lemmatisation. Another alternative is stemming. If you want to learn the difference between these two, I have briefly explained the difference here.
Lemmatisation helps convert words to their dictionary form. Taking our previous example, let’s lemmatise them and see how the output looks like:
lemmatiser = WordNetLemmatizer()for word in ['play', 'plays', 'playing', 'played']:    print(lemmatiser.lemmatize(word, 'v'))
Cool, all words are now converted to ‘play’. Notice how we passed ‘v’ as the second argument in lemmatize method? The ‘v’ is a part-of-speech tag. Accurate lemmatisation, in fact, depends on the part-of-speech (POS) tag we provide to the lemmatiser along with the word. To demonstrate this, let’s rerun the previous script with one small change where ‘v’ becomes ‘n’:
lemmatiser = WordNetLemmatizer()for word in ['plays', 'playing', 'played']:    print(lemmatiser.lemmatize(word, 'n'))
This time, not all words have been converted to play. This brief example shows how a part-of-speech tag can impact the effectiveness of the lemmatisation. So what exactly is a part-of-speech tag? Simply put, it refers to a grammatical category of the the word. The word ‘movie’ is a noun and ‘watch’ can be either a verb or a noun depending on the context. Nouns and verbs are examples of part-of-speech. In the first run, we told the lemmatiser that the words provided are verbs (hence ‘v’) and in the second run, nouns (hence ‘n’). Using nltk’s part-of-speech tagger: pos_tag(), we can tag each word with its part-of-speech.
We will shortly create a custom function that does part-of-speech tagging and lemmatisation based on the tags. We will pass this function to the analyzer parameter of TdidfVectoriser(). When we do this, some of the parameters used previously such as token_pattern, stop_words would no longer be supported. Thus, we have to also include a tokenising step in our custom function:
def preprocess_text(text):    # 1. Tokenise to alphabetic tokens    tokeniser = RegexpTokenizer(r'[A-Za-z]+')    tokens = tokeniser.tokenize(text)        # 2. POS tagging    pos_map = {'J': 'a', 'N': 'n', 'R': 'r', 'V': 'v'}    pos_tags = pos_tag(tokens)        # 3. Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    tokens = [lemmatiser.lemmatize(t.lower(), pos=pos_map.get(p[0], 'v')) for t, p in pos_tags]return tokens
One thing to note here is that pos_tag() and lemmatiser.lemmatize() use different naming conventions for part-of-speech tags, so we had to map the part-of-speech produced by the nltk tagger to the names that the lemmatiser can understand. That’s why we have pos_map. Now, let’s preprocess the data and assess:
vectoriser = TfidfVectorizer(analyzer=preprocess_text, min_df=30, max_df=.7)inspect(vectoriser, X_train)
It reduced the number of columns to 10,754. That’s about 16% reduction in the number of columns compared to 12,805 columns in simpler approach. It took about 11 minutes on my computer which is ~85 times slower. If we think about it, lemmatisation is not going to alter every word in a review. Let’s take ‘The movie was awesome.’ sentence as an example. The only word would benefit from a lemmatisation is ‘was’. So keeping this in mind, sometimes it’s fine to use a default part-of-speech for all words if you need to do lemmatisation quicker. In that case, our custom function simplifies to:
# Simple approachdef preprocess_text(text):    # 1. Tokenise to alphabetic tokens    tokeniser = RegexpTokenizer(r'[A-Za-z]+')    tokens = tokeniser.tokenize(text)        # 2. Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    tokens = [lemmatiser.lemmatize(t.lower(), pos='v') for t in tokens]return tokensvectoriser = TfidfVectorizer(analyzer=preprocess_text, min_df=30, max_df=.7)inspect(vectoriser, X_train)
The number of columns is very close to the version we used part-of-speech tagger but it only took about a minute. So between these two versions, I prefer using the default part-of-speech tag over the other for this text dataset. Of note, if we don’t provide a value to the pos argument, the lemmatiser defaults to ’n’. Generally speaking, I usually like to use ‘v’ as a default because I find that it normalises more words in general. However, the more suitable default pos will depend on the data.
In this approach, we will further clean up the data. Based on the learnings from the exploratory data analysis and general preprocessing ideas, here’s what we’ll do:
◼ Correct typos: ‘chracter’ to ‘character’◼ Convert British spelling to American spelling: ‘realise’ to ‘realize’◼ Remove stop words
def convert_to_american(token):    # Copied from here    token = re.sub(r"(...)our$", r"\1or", token)    token = re.sub(r"([bt])re$", r"\1er", token)    token = re.sub(r"([iy])s(e$|ing|ation)", r"\1z\2", token)    token = re.sub(r"ogue$", "og", token)    return tokendef correct_typo(tokens):    spell = SpellChecker()    return [spell.correction(t) if len(spell.unknown([t]))>0 else t for t in tokens]        def preprocess_text(text):    # 1. Tokenise to alphabetic tokens    tokeniser = RegexpTokenizer(r'[A-Za-z]+')    tokens = tokeniser.tokenize(text)        # 2. Lowercase and lemmatise    lemmatiser = WordNetLemmatizer()    tokens = [lemmatiser.lemmatize(t.lower(), pos='v') for t in tokens]# 3. Correct spelling (this won't convert 100% )    tokens = correct_typo(tokens)        # 4. Convert British spelling to American spelling (this won't convert 100%)    tokens = [convert_to_american(t) for t in tokens]# 5. Remove stopwords    stop_words = stopwords.words('english')    stop_words.extend(['cannot', 'could', 'done', 'let', 'may' 'mayn',  'might',  'must', 'need', 'ought', 'oughtn',                        'shall', 'would', 'br'])    tokens = [t for t in tokens if t not in stop_words]        return tokens
In addition to these, we could keep adding other layers to correct and clean up. However, each additional step will add complexity and increase run time without a guaranteed improvement in model performance. Here are some ideas you could try in general when preprocessing text which is not particularly useful in our case but could be useful in other cases:
◼ Clean up junks such as html tags, email addresses, URLs ◼ Convert numbers to words instead of dropping them◼ Convert emojis or emoticons to words
Alright, let’s preprocess and inspect the output:
vectoriser = TfidfVectorizer(analyzer=preprocess_text, min_df=30, max_df=.7)inspect(vectoriser, X_train)
A few hours after I started running this code, it was still running. As I deemed it was taking too long, I had to interrupt the kernel to stop the query. To see how much slower this approach takes relative to the previous two, I reduced the size of the dataset to 1/9th of its size with the script below:
train = train.sample(5000, random_state=seed)
And I have ran all three approaches on this smaller subset of data. Here’s the comparison of the three approaches:
The less simple approach standardised the tokens more than the other two but at a really high cost. It takes at least 1000 times longer to preprocess compared to the others. There is a chance that this ratio could look worse when the dataset grows. Hence, it’s not practical to pursue less simple approach further unless it’s optimised to run faster.
👂 If you have a better computer than mine, this run time concern may not necessarily be a constraint for you in which case feel free to pursue further.
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. Hopefully, you have learned a few different practical ways to preprocess text that you could apply to your next NLP project. In the next post, we will build a sentiment classifier. Here are links to the other two posts of the series:◼️ Exploratory text analysis in Python◼️ Sentiment classification in Python
Here are links to the my other NLP-related posts:◼️ Simple wordcloud in Python(Below lists a series of posts on Introduction to NLP)◼️ Part 1: Preprocessing text in Python◼️ Part 2: Difference between lemmatisation and stemming◼️ Part 3: TF-IDF explained◼️ Part 4: Supervised text classification model in Python◼️ Part 5A: Unsupervised topic model in Python (sklearn)◼️ Part 5B: Unsupervised topic model in Python (gensim)
Bye for now 🏃💨