Logic and implementation of a spam filter machine learning algorithm | by Lily Chen | Towards Data Science

Below is an email caught by Gmail’s spam filter. How did the spam filter decide this was a spam?
In this article, I talk about Naive Bayes algorithm for spam classification.
We know all of the words that this email contains:
[want, to, maximize, your, time, ..., save, on, the, breakers, bonus]
At the high level, the way spam classification works is this:
What’s the probability this email is spam given the list of words in the email?
The mathematical way of representing this question is:
We need to find this probability.
Let’s break it down.
One of the words in the email is “bonus”.
If bonus was the only word in the email, what would be the probability this email is spam?
Let’s solve for P(spam|bonus) .
In order to find P(spam|bonus), you need an existing dataset of emails. This is your training data. The probability that an email is spam is based on information from this training data.
Let’s say you have an existing dataset of 100 emails. 20 are spam and 80 are not spam. The spam emails combined contain 2000 words. The non spam emails combined contain 8000 words. Of the spam emails, the word “bonus” appears 1000 times. Of the non spam emails, the word bonus appears 800 times. Below is what your data looks like:
You can think of area of the rectangle as number of words.
We need to find probability of “spam” given “bonus”, or P(spam | bonus) .
To answer this question, we use Bayes theorem.
P(spam):
The probability of spam is number of spam divided by total number of emails.
20/100 = 0.2
P(bonus|spam):
The probability of seeing “bonus” in spam is number of “bonus” in spam divided by total number of words in spam.
1000/2000 = 0.5
P(bonus):
Probability of bonus is total number of “bonus” (1000 in spam and 800 in non spam) divided by total number of words:
(1000 + 800)/10000 = 0.18.
This is the same as the area taken by “bonus” in the entire rectangle.
P(spam|bonus):
Plug in the numbers into Bayes theorem to find probability an email is spam if it contains “bonus”.
If “bonus” was the only word in the email, the probability that the email is spam is 0.56.
Let’s look at 2 words now. What if the email only contains: “save bonus”?
Instead of P(spam|bonus), we need P(spam|save, bonus).
Using Bayes theorem, this becomes:
P(spam):
The probability of spam is still number of spam divided by total number of emails.
20/100 = 0.2
P(save, bonus|spam):
By chain rule of probability
We already calculated P(bonus|spam) to be 0.5 because there are 1000 “bonus” words out of 2000 total words in spam emails.
Let’s say there are 100 “save” in spam emails and 160 “save” in non spam emails. Our rectangle of words now look like this:
P(save|spam) is going to be number of “save” in spam divided by total number of words in spam.
P(save|spam) = 100/2000 = 0.05
Taken together,
P(save, bonus|spam) = P(save|spam)*P(bonus|spam) = 0.5 * 0.05 = 0.025
Note: this formula assumes the words in the email are conditionally independent of each other. In other words, having “save” in the email does not change the probability of having “bonus” in the same email, and vice versa. This assumption is why we call the algorithm Naive Bayes.
P(save, bonus):
To find what is P(save, bonus), we need to do some math.
We know that P(spam|save, bonus) + P(not spam|save, bonus) must be 1.
Substitute in Bayes equation:
Multiply both sides by P(save, bonus) to get:
P(not spam) is just number of non spam emails divided by total number of emails. P(not spam) = 800/1000, or 0.8.
P(save, bonus | not spam) = P(save |not spam) * P(bonus|not spam) = 160/8000 * 800/8000 = 0.002
P(save, bonus) = 0.2*0.025 + 0.8*0.002 = 0.0066
P(spam|save, bonus):
Plug in the numbers into Bayes theorem to find probability an email is spam if it contains “bonus”.
Now, let’s analyze the real Bay to Breakers of 130+ words.
Applying the same logic for more words, the formula becomes
P(spam):
Probability of spam is still 20/100, or 0.2.
P(word_1, word_2,...,word_n| spam):
We find P(word|spam) for every word, and take the product of all the probabilities.
P(word_1, word_2,...,word_n):
The denominator of the equation, i.e. P(word_1, word_2,...,word_n), can also be written as
P(spam) * P(word_1, word_2,...,word_n| spam) + P(not spam) * P(word_1, word_2,...,word_n | not spam)
The first half of the equation is basically the nominator of Bayes theorem, and solved in the steps above.
P(not spam) is 80/100. To find P(word_1, word_2, ..., word_n | not spam), we find P(word|not spam) for every word, and take the product of all the probabilities.
Mathematically, it is equivalent to
Taken everything together, the probability that the email from Bay to Breakers is spam is:
Everything works great....except if you encounter a new word previously unseen in your training data. Let’s say an email contains the single word “GetYourFreeCookieNow” and your algorithm needs to decide whether it’s spam or not.
Most likely your training data does not include this word. In our example, "GetYourFreeCookieNow” was neither present in our 20 spam email nor 80 non spam email. If that’s the case, P(GetYourFreeCookieNow | spam) is zero. P(GetYourFreeCookieNow | not spam) is also zero. You can’t divide 0 by 0, so the algorithm will fail.
Similarly, if a word (let’s say “calendar”) never appeared in spam emails of your training set, you always get probability of 0 for spam if an email contains “calendar”.
So let’s tweak our Naive Bayes algorithm to fix this problem.
We solve this problem by adding a count of 1 to each word. For example, even though “GetYourFreeCookieNow” was not in our training data, the nominator of P(GetYourFreeCookieNow | spam) will now be 1 instead of 0. Therefore, you will not get a zero for P(GetYourFreeCookieNow | spam).
Let’s look at our P(bonus|spam) example.
Before adding 1 to each word:
After adding 1 to each word:
W_unique is the number of unique words in all of the emails.
Below is a video illustration of spam classification based on Naive Bayes.
Now that we talked about the theory behind email spam classification. Let’s implement it.
First, you need a training set. This is the data I used from Kaggle. It’s in a csv format and looks like this:
The label “ham” means “not spam”.
From this data, we need to compute several things:
Number of spam emailsNumber of non spam emails (i.e. “ham” emails).Number of unique words in the vocabulary.Total number of words in spam emails.Total number of words in ham emails.Number of occurrences of each word in spam emails.Number of occurrences of each word in ham emails.
Number of spam emails
Number of non spam emails (i.e. “ham” emails).
Number of unique words in the vocabulary.
Total number of words in spam emails.
Total number of words in ham emails.
Number of occurrences of each word in spam emails.
Number of occurrences of each word in ham emails.
I’m going to iterate through the csv file email by email, and increment the value for the 7 variables we need. You can find the code in my Github repo.
github.com
This is the result I got:
number of spam emails: 1451number of non spam emails: 3512total unique words: 45819total spam words: 282332total non spam words: 487196spam words with count: {'subject': 1741, 'photoshop': 80, 'windows': 207, 'office': 204, 'cheap': 79, 'main': 17, 'trending': 2, ... }non spam words with count: {'subject': 6111, 'enron': 6209, 'methanol': 114, 'meter': 2381, 'this': 4680, 'is': 4730, 'a': 5765, 'follow': 93, ...}
Now, we’re ready to classify whether the Bay to Breakers email is spam.
There are a few things about my algorithm that differ from the formula presented above.
Calculate LOG of probability
Calculate LOG of probability
I calculated the log of the probability instead of the probability itself. This is because the product of P(word | spam) for all the words is going to be very close to 0, and the computer just round it down to zero.
Just a reminder, Bayes theorem is this:
We do a transformation by applying the log. This is the new nominator:
To calculate the nominator of spam, my algorithm iterate through each word in the email, and sum up the log of P(word | spam).
The complete code is in this Github repo.
2. Compare the nominator of Bayes theorem for probability of spam and probability of not spam.
We don’t need to compute the denominator of Naive Bayes theorem at all. We just need to compare which nominator is greater: the probability of spam given the words in the email, or the probability of not spam given those words.
Whichever is greater (spam or not spam), we’ll categorize the email in that category.
Using these 2 tweaks, my algorithm indeed found that the Bay to Breakers email is SPAM!
These are the print statements when running my algorithm:
log_p_spam:  -1.229757422703253log_p_ham:  -0.3458247208324336log_p_words_given_spam:  -1028.860861014028log_p_words_given_ham:  -1051.386635753655Is Spam? True
Note: This may not be how Gmail spam filter actually works. This is a Naive Bayes algorithm, and works relatively well according to research. You can also use logistic regression for spam classification. I will talk about comparison of different algorithms and their tradeoffs in later articles. Stay tuned!