How to build and apply Naive Bayes classification for spam filtering | by Pavlo Horbonos | Towards Data Science

I believe, that today almost everyone has a smartphone and a lot of people keep an email or two. This means that you have to be familiar with tons of messages proposing a lot of money, fantastic lottery wins, great presents and secrets of life. We get dozens of spam messages every day unless you use well-trained filters. They may be harmful, just annoying or space-consuming, but they also can contain viruses or fishing attempts. In any case, it is not the content we want to deal with. So the demand for good spam filters is always high.
Let me acquaint you (or remind if you are already familiar) with one of the pretty effective algorithms for spam filtering: Naive Bayes classification. Although there is already existing implementation in scikit-learn package, I want to recreate the algorithm from scratch. Firstly, I want to uncover the logic hidden behind the implementation. And secondly, I want to show the algorithm in conjunction with the dataset preparation.
Naive Bayes classification is a simple probability algorithm based on the fact, that all features of the model are independent. In the context of the spam filter, we suppose, that every word in the message is independent of all other words and we count them with the ignorance of the context.
Our classification algorithm produces probabilities of the message to be spam or not spam by the condition of the current set of words. Calculation of the probability is based on the Bayes formula and the components of the formula are calculated based on the frequencies of the words in the whole set of messages.
So, how does the algorithm work?
First of all, we take the Bayes formula of the conditional probability and apply it to our task:
The probability of the message, that contains words (w1, w2, w3, ...) to be spam is proportional to the probability to get the spam multiplied by a product of probabilities for every word in the message to belong to a spam message. What does it mean? For every word in the message, we calculate a probability of it to be found in spam. In our context:
P_spam — the part of spam messages in our dataset
P_wi_spam — the probability of a word to be found in the spam messages.
By the same logic we define:
P_not_spam — the part of non-spam messages in the dataset
P_wi_non_spam — the probability of a word to be found in the non-spam messages.
But we still do not know how to calculate each word probabilities. Though, we have another formula:
What do we have here:
N_vocabulary — the number of unique words in the whole dataset.
N_spam — the total number of words in the spam messages.
N_wi_spam — the number of a word repeats in all spam messages.
Alpha — the coefficient for the cases when a word in the message is absent in our dataset.
Shortly: the probability of a word to belong to the spam message is a frequency of this word in the “spam part” of our dataset.
Also, the same formula (but with other values) is correct for the probability of a word to belong to non-spam messages.
The math is over, but do not worry — I will show all the values and formulas on the dataset example.
For our purpose, we will use the collection of SMS messages, which was put together by Tiago A. Almeida and José María Gómez Hidalgo. It is free and can be downloaded from the UCI Machine Learning Repository.
Dataset structure is simple. It contains two columns — one for the label “spam/ham” and another for the text of the message.
It contains 5572 records of different messages together with 747 spam messages.
sms_data.groupby('Label').count()Out[6]:ham 4825spam 747
We have enough data to start the analysis.
Before the algorithm application, we need to prepare data. First of all, we will remove the punctuation. Then we will convert all the text into the lower-case and split it into the separate words.
The next thing is to split the dataset to train and test data. But we also need to keep the distribution of spam and non-spam messages.
And at last, we have to prepare the vocabulary and count the number of separate words in each message.
Concat the resulting table of words count to our train data:
And the data is ready.
We will follow the formulas mentioned above and define the main values:
probability of message to be spam
Pspam = train_data[‘Label’].value_counts()[‘spam’] / train_data.shape[0]
probability of non-spam message
Pham = train_data[‘Label’].value_counts()[‘ham’] / train_data.shape[0]
the number of words in spam messages
Nspam = train_data.loc[train_data[‘Label’] == ‘spam’,                        ‘SMS’].apply(len).sum()
the number of words in non-spam messages
Nham = train_data.loc[train_data[‘Label’] == ‘ham’,                      ‘SMS’].apply(len).sum()
the size of the vocabulary
Nvoc = len(train_data.columns - 3)
set alpha = 1
To complete the formula we will define the functions to determine the probabilities of the given word to belong to spam and non-spam messages:
Our classification function:
After the function application, we see impressive results on the test data: 99.1% of data were successfully classified.
Just for the interest let’s look on the incorrectly recognized messages:
Now you are familiar (or freshened up in the memory) with one of the most effective spam-classification algorithms. I hope it was helpful and descriptive. You can find the Jupyter notebook with working example on my GitHub:
github.com
You are welcome to share other algorithms called upon the fight with spam.