Beginner’s Introduction to NLP — Building a Spam Classifier | by Jason Chong | Towards Data Science

Words, sentences, paragraphs and essays. We use them almost every day of our adult lives. Whether you are sending out a tweet, composing an email to your colleague, or writing an article like what I am doing now, as humans, we all use words to communicate our thoughts and our ideas.
Now, imagine a world where we could teach computers to interact with words the same way that we would with another human being. Imagine a world where computers are capable of not only detecting human speech but more importantly, capable of learning the nuances of that speech and as a result, derive the underlying meaning or intent of a given message.
That is precisely what natural language processing (NLP) is all about.
NLP refers to the field within artificial intelligence that deals with the interaction between computers and humans via the natural language. This includes enabling computers to manipulate, analyse, interpret as well as generate human language.
I recently had some time to dabble into some basics of NLP and I wanted to use this article as means of not only sharing what I have learned but also further reinforce my understanding of the topic. Specifically, this article was inspired by NLP with Python for Machine Learning Essential Training, a LinkedIn course given by Derek Jedamski.
I would like to hereby give full credits and reference to Derek for his work and I highly recommend checking out his other courses on machine learning in Python.
My goal with this article is to recreate the main project that was introduced in the course, a spam (binary) classifier built using the SMS spam collection dataset which contains a set of 5,572 SMS messages in English. Once ready, the classifier will be able to read a given string of text and subsequently classify the text as either ham or spam.
In this article, I will walk through the steps of building out the spam classifier and along the way, highlight some key concepts in natural language processing. If you would like to follow along, the code for this particular project can be found on my GitHub here.
Without further ado, here we go!
Like any professional data scientist, let’s begin by first exploring the dataset.
Exploratory data analysis is the process of performing an initial investigation on data to discover patterns, spot anomalies, test hypothesis and check assumptions with the help of summary statistics and graphical representations (definition credits to Prasad Patil on “What is Exploratory Data Analysis”).
data = pd.read_csv("spam.csv", encoding = "latin-1")data = data.dropna(how = "any", axis = 1)data.columns = ['label', 'body_text']data.head()
The input data has 5,572 rows (each row represents a unique text message) and 2 columns: label and body_text. Out of 5,572 rows, 747 of them are spam and the remaining 4,825 are ham. We also have no missing data, yay!
Let’s use seaborn for a more visual representation of our input data.
total = len(data)plt.figure(figsize = (5, 5))plt.title("Number of spam vs ham messages")ax = sns.countplot(x = 'label', data = data)for p in ax.patches:    percentage = '{0:.0f}%'.format(p.get_height() / total * 100)    x = p.get_x() + p.get_width() / 2    y = p.get_height() + 20    ax.annotate(percentage, (x, y), ha = 'center')plt.show()
As we can see, there are significantly more ham messages (87%) than there are spam messages (13%).
Technically speaking, at this stage, we should be concerned about issues related to an unbalanced dataset. However, for the purpose of keeping things simple, let’s just ignore this for now.
Feature engineering is the process of creating new features and/or transforming existing features to get the most out of our data.
In this section, we will look to create two new features:
body_len: length of body text excluding spaces
punct%: percentage of punctuation in the body text
# body_lendata['body_len'] = data.body_text.apply(lambda x: len(x) - x.count(" "))# punct%def count_punct(text):    count = sum([1 for char in text if char in string.punctuation])    return round(count/(len(text) - text.count(" ")), 3) * 100 data[‘punct%’] = data.body_text.apply(lambda x: count_punct(x))
Now, we can proceed to use the newly created features to explore the distribution of our input data.
bins = np.linspace(0, 200, 40)data.loc[data.label == 'spam', 'body_len'].plot(kind = 'hist', bins = bins, alpha = 0.5, density = True, label = 'spam')data.loc[data.label == 'ham', 'body_len'].plot(kind = 'hist', bins = bins, alpha = 0.5, density = True, label = 'ham')plt.legend(loc = 'best')plt.xlabel("body_len")plt.title("Body length ham vs spam")plt.show()
As we can see, spam messages have a longer body length i.e. contain more words compared to ham messages.
Now, what about punctuation percentage?
bins = np.linspace(0, 50, 40)data.loc[data.label == 'spam', 'punct%'].plot(kind = 'hist', bins = bins, alpha = 0.5, density = True, label = 'spam')data.loc[data.label == 'ham', 'punct%'].plot(kind = 'hist', bins = bins, alpha = 0.5, density = True, label = 'ham')plt.legend(loc = 'best')plt.xlabel("punct%")plt.title("Punctuation percentage ham vs spam")plt.show()
Here, the two distributions look pretty similar although ham messages do appear to have a longer tail i.e. ham messages tend to have a higher punctuation percentage.
If you have any experience working with real-world data, you would know that real-world data are oftentimes extremely messy and not so straightforward to process. This one is no exception.
There is a lot of work to be done before we can move on to actually modelling and predicting the data. More specifically, in order for us to better manage the messy text messages, we will need to perform the following pre-processing steps (these steps are very specific to the NLP pipeline):
Convert words into lowercase letters
Remove punctuation and stopwords
Tokenisation
Stemming vs lemmatisation (text normalisation)
Let’s investigate each step one by one.
Python does not see all characters as equal. For consistency, we will need to convert all words into lowercase letters. For example, Python will return False to the first statement below but will return True to the second statement.
# False statement "NLP" == "nlp"# True statement "NLP".lower() == "nlp"
Similar to the argument made above, the rationale for removing punctuation is because punctuation does not hold any meaning in a text. Thus, we want Python to only focus on the words in a given text and not worry about the punctuations that are involved.
Let’s take the following as an example.
# False statement"I love NLP" == "I love NLP."
As humans, we can immediately see that the two texts above are almost exactly identical, except the second one has a period at the end of the sentence.
However, Python will fail to distinguish between the two texts. For that reason, it is important that we remove all punctuation in a sentence in order to allow Python to interpret the text more clearly.
You can find a list of punctuations that are stored in the string library in Python by typing:
string.punctuation
And to remove punctuation in a sentence, we can deploy list comprehension as follows:
# Original text text = 'OMG! Did you see what happened to her? I was so shocked when I heard the news. :('# List comprehension to remove punctuation text = "".join([word for word in text if word not in string.punctuation])
In the context of NLP, tokenisation means to a string or a sentence into a list of characters and we can accomplish this by utilising the regular expression (re) library in Python.
The two most straightforward commands in the library include:
re.split
re.findall
If you are interested to know how they work, I encourage checking out my notebook for more details.
Stopwords are commonly used words in the English language like but, if and the that don’t contribute much to the overall meaning of a sentence. Therefore, stopwords are usually removed in order to reduce the number of tokens Python needs to store and process when building our machine learning model.
Stopwords are stored in nltk.corpus.stopwords which can be accessed as follows:
stopwords = nltk.corpus.stopwords.words('english')stopwords
To remove stopwords in a given string, we can again apply list comprehension.
# Original text text = 'OMG Did you see what happened to her I was so shocked when I heard the news'print(text)# Convert text into list of words in lowercase lettersprint(text.lower().split())# List comprehension to remove stopwords print([word for word in text.lower().split() if word not in stopwords]
After running the code snippet above, the following stopwords will be removed.
did
you
what
to
her
i
was
so
when
the
Notice that we first turned the original text into a list of lowercase words before running our list comprehension. This is because words are stored in lowercase letters in the nltk library.
Stemming: the process of reducing inflection or derived words to their word stem or root by crudely chopping off the ends of a word to leave only the base.Lemmatising: the process of grouping together inflected forms of a word so they can be analyzed as a single term.
Broadly speaking, both stemming and lemmatising serve the purpose of condensing the variations of the same word down to their root form. This is to prevent the computer from storing every single unique word it sees in a corpus of words but instead only take note of a word in its most basic form and correlate other words with similar meanings.
For example, grew, grown, growing and growth are all simply variations to the word grow. In this case, the computer only needs to remember the word grow and not the rest.
To access stemmer and lemmatiser:
ps = nltk.PorterStemmer()wn = nltk.WordNetLemmatizer()
Stemmer takes a more crude approach than lemmatiser by simply chopping off the end of a word using heuristics, without any knowledge of the context in which a word is used. As a result, stemmer can sometimes not return an actual word in the dictionary.
Lemmatiser, on the other hand, will always return a dictionary word. Lemmatiser considers multiple factors before simplifying a given word and is generally more accurate. However, this comes at the cost of being slower and more computationally expensive than stemmer.
Awesome, now that we understand all the pre-processing steps that go into text cleaning, we want to summarise everything into a single function called clean_text that we can then apply to our input data.
# Create function for text cleaning def clean_text(text):    text = "".join([word.lower() for word in text if word not in string.punctuation])    tokens = re.findall('\S+', text)    text = [wn.lemmatize(word) for word in tokens if word not in stopwords]    return text# Apply function to body_text data['cleaned_text'] = data['body_text'].apply(lambda x: clean_text(x))data[['body_text', 'cleaned_text']].head(10)
From there, we can compute the most common words that are observed in ham versus spam messages.
Vectorisation is the process of encoding text as integers to create feature vectors.
In this section, we will look at three different functions for vectorising text in scikit-learn:
CountVectorizer
TfidfTransformer
TfidfVectorizer
CountVectorizer creates a document-term matrix where the entry of each cell will be a count of the number of times that word occurred in that document.
TfidfTransformer is similar to that of a CountVectorizer but instead of the cells representing the count, the cells represent a weighting that is meant to identify how important a word is to an individual text message. The formula to compute the weighting for each cell is as follows:
To demonstrate this, let’s look at an example.
# CountVectorizercorpus = ['I love bananas', 'Bananas are so amazing!', 'Bananas go so well with pancakes']count_vect = CountVectorizer()corpus = count_vect.fit_transform(corpus)pd.DataFrame(corpus.toarray(), columns = count_vect.get_feature_names())
Each row in the dataframe represents a sentence (document) and each column represents a unique word, excluding stopwords, in the entire corpus. For instance, “Bananas are so amazing” have values of 1 (0 in the others) in the amazing, are, bananas and so columns because each one of those words shows up once in that particular sentence.
TfidfTransformer, on the other hand, works as follows:
# TfidfTransformertfidf_transformer = TfidfTransformer()corpus = tfidf_transformer.fit_transform(corpus)pd.DataFrame(corpus.toarray(), columns = count_vect.get_feature_names())
Recall, the cells in TF-IDF represent a weighting of how important a word is to an individual text message.
Let’s take the bananas column as an example. While the word bananas show up only once in each of the three sentences, a higher weighting is assigned to the first sentence compared to the second and third because the first sentence has the shortest length. In other words, the word bananas is more important in the first sentence than it is in the second and the third.
The rarer (more infrequent) a word is in a document or corpus, the higher the weighting under TF-IDF.
TfidfVectorizer is equivalent to CountVectorizer followed by TfidfTransformer.
# TfidfVectorizercorpus = ['I love bananas', 'Bananas are so amazing!', 'Bananas go so well with pancakes']tfidf_vect = TfidfVectorizer()corpus = tfidf_vect.fit_transform(corpus)pd.DataFrame(corpus.toarray(), columns = tfidf_vect.get_feature_names())
As we can see, the result is exactly the same. Therefore, for convenience, we will use TfidfVectorizer for our project.
Finally, time for some fun!
Now that our data is ready, we can finally move on to modelling, that is actually building our spam filter to classify a given text as ham or spam.
Here, we will consider two approaches to modelling: train-test-split and pipeline as well as two types of machine learning models or more specifically, ensemble methods: random forest and gradient boosting.
If you are new to machine learning, an ensemble method is essentially a technique whereby multiple models are created and combined with the goal of producing better prediction accuracy than any of the single models individually.
# Train test splitX_train, X_test, Y_train, Y_test = train_test_split(data[['body_text', 'body_len', 'punct%']], data.label, random_state = 42, test_size = 0.2)# Instantiate and fit TfidfVectorizertfidf_vect = TfidfVectorizer(analyzer = clean_text)tfidf_vect_fit = tfidf_vect.fit(X_train['body_text'])# Use fitted TfidfVectorizer to transform body text in X_train and X_testtfidf_train = tfidf_vect.transform(X_train['body_text'])tfidf_test = tfidf_vect.transform(X_test['body_text'])# Recombine transformed body text with body_len and punct% featuresX_train = pd.concat([X_train[['body_len', 'punct%']].reset_index(drop = True), pd.DataFrame(tfidf_train.toarray())], axis = 1)X_test = pd.concat([X_test[['body_len', 'punct%']].reset_index(drop = True), pd.DataFrame(tfidf_test.toarray())], axis = 1)
RandomForestClassifier is an ensemble learning method that utilises bagging to construct a collection of decision trees and then aggregates the predictions of each tree to determine the final prediction.
The key hyperparameters for RandomForestClassifier include:
max_depth: maximum depth of each decision tree
n_estimators: how many parallel decision trees to build
random_state: for reproducibility purpose
n_jobs: number of jobs to run in parallel
Before we begin training our random forest model on the data, let’s first build a manual grid search, using nested for loops, to find the most optimal set of hyperparameters.
def explore_rf_params(n_est, depth):    rf = RandomForestClassifier(n_estimators = n_est, max_depth = depth, n_jobs = -1, random_state = 42)    rf_model = rf.fit(X_train, Y_train)    Y_pred = rf_model.predict(X_test)    precision, recall, fscore, support = score(Y_test, Y_pred, pos_label = 'spam', average = 'binary')    print(f"Est: {n_est} / Depth: {depth} ---- Precision: {round(precision, 3)} / Recall: {round(recall, 3)} / Accuracy: {round((Y_pred==Y_test).sum() / len(Y_pred), 3)}")    for n_est in [50, 100, 150]:    for depth in [10, 20, 30, None]:        explore_rf_params(n_est, depth)
From the output above, we can conclude that:
Precision is constant at 1 for all cases
Recall and accuracy both improve as max_depth increases with None giving the best results
There is very little improvement adding more trees after the 100th tree so we will set n_estimators = 100.
Now that we have our hyperparameters, we can proceed to fit our model to the data.
# Instantiate RandomForestClassifier with optimal set of hyperparameters rf = RandomForestClassifier(n_estimators = 100, max_depth = None, random_state = 42, n_jobs = -1)# Fit modelstart = time.time()rf_model = rf.fit(X_train, Y_train)end = time.time()fit_time = end - start# Predict start = time.time()Y_pred = rf_model.predict(X_test)end = time.time()pred_time = end - start# Time and prediction resultsprecision, recall, fscore, support = score(Y_test, Y_pred, pos_label = 'spam', average = 'binary')print(f"Fit time: {round(fit_time, 3)} / Predict time: {round(pred_time, 3)}")print(f"Precision: {round(precision, 3)} / Recall: {round(recall, 3)} / Accuracy: {round((Y_pred==Y_test).sum() / len(Y_pred), 3)}")
Fit time: 15.684
Predict time: 0.312
Precision: 1.0
Recall: 0.833
Accuracy: 0.978
Alternatively, we can also use a confusion matrix to visualise the result of a binary classification.
# Confusion matrix for RandomForestClassifiermatrix = confusion_matrix(Y_test, Y_pred)sns.heatmap(matrix, annot = True, fmt = 'd')
GradientBoostingClassifier, on the other hand, is also an ensemble learning method that takes an iterative approach, known as bagging, to combine weak learners to create a strong learner by focusing on mistakes of prior iterations.
The key hyperparameters for GradientBoostingClassifier include:
learning_rate: weight of each sequential tree on the final prediction
max_depth: maximum depth of each decision tree
n_estimators: number of sequential trees
random_state: for reproducibility purpose
Unfortunately, grid search for gradient boosting will take a long time so I have decided to stick with the default hyperparameters for now.
# Instantiate GradientBoostingClassifiergb = GradientBoostingClassifier(random_state = 42)# Fit modelstart = time.time()gb_model = gb.fit(X_train, Y_train)end = time.time()fit_time = end - start# Predict start = time.time()Y_pred = gb_model.predict(X_test)end = time.time()pred_time = end - start# Time and prediction resultsprecision, recall, fscore, support = score(Y_test, Y_pred, pos_label = 'spam', average = 'binary')print(f"Fit time: {round(fit_time, 3)} / Predict time: {round(pred_time, 3)}")print(f"Precision: {round(precision, 3)} / Recall: {round(recall, 3)} / Accuracy: {round((Y_pred==Y_test).sum() / len(Y_pred), 3)}")
Fit time: 262.863
Predict time: 0.622
Precision: 0.953
Recall: 0.813
Accuracy: 0.97
A pipeline chains together multiple steps in the machine learning workflow where the output of each step is used as input to the next step.
# Instantiate TfidfVectorizer, RandomForestClassifier and GradientBoostingClassifier tfidf_vect = TfidfVectorizer(analyzer = clean_text)rf = RandomForestClassifier(random_state = 42, n_jobs = -1)gb = GradientBoostingClassifier(random_state = 42)# Make columns transformertransformer = make_column_transformer((tfidf_vect, 'body_text'), remainder = 'passthrough')# Build two separate pipelines for RandomForestClassifier and GradientBoostingClassifier rf_pipeline = make_pipeline(transformer, rf)gb_pipeline = make_pipeline(transformer, gb)# Perform 5-fold cross validation and compute mean score rf_score = cross_val_score(rf_pipeline, data[['body_text', 'body_len', 'punct%']], data.label, cv = 5, scoring = 'accuracy', n_jobs = -1)gb_score = cross_val_score(gb_pipeline, data[['body_text', 'body_len', 'punct%']], data.label, cv = 5, scoring = 'accuracy', n_jobs = -1)print(f"Random forest score: {round(mean(rf_score), 3)}")print(f"Gradient boosting score: {round(mean(gb_score), 3)}")
Random forest score: 0.973
Gradient boosting score: 0.962
While both models, in this particular example, have returned very similar prediction results, it is important to bear in mind the trade-offs that may occur in other scenarios where this is not the case. More specifically, it is worth considering the business context and the overall purpose for which the model is built.
For example, in spam classification, it is better to optimise for precision as we can probably deal with some spam messages in our inbox here and there but we definitely don’t want our model to classify an important message as spam. In contrast, in fraud detection, it is much better to optimise for recall as it is more costly if our model fails to identify a real threat (false negative) than it is if it identifies a false threat (false positive).
To wrap up, in this article, we have looked at an end-to-end natural language processing (NLP) project which involves building a binary classifier capable of classifying a given text message as spam or ham.
We started off by exploring the dataset, followed by feature engineering where we created two new features: body_len and punct%. We then moved on to several preprocessing steps that are specific to the NLP workflow such as:
Convert words into lowercase letters
Remove punctuation and stopwords
Tokenisation
Stemming vs lemmatisation (text normalisation)
After that, we performed vectorisation using in order to encode text and turn them into feature vectors for machine learning. Finally, we finished off by building two separate prediction models, random forest and gradient boosting, as well as compare their respective accuracy and overall model performance.
With that said, thank you so much for reading and I look forward to seeing you in my next article!