Learn Bayes Theorem by Detecting SPAM | by GreekDataGuy | Towards Data Science

This tutorial has 2 parts:1. Deriving Bayes theorem from conditional probability2. Predicting if an SMS message is a spam
I covered conditional probability in more depth here.
Conditional probability tells us the probability of an event occurring, given another event.
P(A|B) = P(A ∩ B) / P(B) is the probability A occurs, in cases where we knowB occurs. It’s calculated as the probability that both A and B occur, divided by the probability that B occurs.
But what if we wanted to find the reverse, the probability of B, in cases where A occurs?
Sometimes conditional probability works great for that to. But sometimes it’s easier to use Bayes theorem.
Wikipedia says,
In probability theory and statistics, Bayes’s theorem (alternatively Bayes’s law or Bayes’s rule) describes the probability of an event, based on prior knowledge of conditions that might be related to the event.
We start with the formula for conditional probability which can be written either, “A given B” or “B given A”.
Note that intuitively, P(A∩B) and P(B∩A) are the same (see below). This means we can use them interchangeably. Keep this in mind for later.
We’ll start with the 1st formula, P(A|B)= P(A∩B) / P(B).
Multiple both sides by P(B). This will cancel out the P(B) denominator on the right, leaving us with below.
What we can now see (more easily if we swapped the left and right sides) is that P(A∩B)= P(A|B) * P(B) . We’ll plug this back into our 2nd original formula (original below).
To get this formula (modified below).
Which is Bayes theorem.
We’ll now use Bayes theorem to try and predict spam in SMS messages.
Bayesian inference has a long history in spam detection. We’ll get into the basics here with some real data.
In our case, the probability an SMS is spam, given some word, is equal to the probability of the word, given it is in a spam SMS, multiplied by the probability of spam, all divided by the probability of the word.
Download the dataset from Kaggle, and inspect it in a dataframe.
import pandas as pddf = pd.read_csv('sms-spam.csv', encoding='ISO-8859-1')df.head(3)
The columns in the original CSV don’t make sense. So we’ll move the useful information into 2 new columns, one of which is a boolean indicating if the SMS is spam.
FYI, “ham” means “not spam”.
import numpy as npdf['sms'] = df['v2']df['spam'] = np.where(df['v1'] == 'spam', 1, 0)df.head(3)
Now drop the old columns.
df = df[['sms','spam']]df.head()
Much better.
Check the number of records.
len(df)#=> 5572
That’s a lot. Let’s work with a sample of 25% of the original data.
sample_df = df.sample(frac=0.25)len(sample_df)#=> 1393
That’s better.
Now split the data into 2 separate dataframes, one for spam and one for ham.
spam_df = sample_df.loc[df['spam'] == 1]ham_df = sample_df.loc[df['spam'] == 0]print(len(spam_df))print(len(ham_df))#=> 180#=> 1213
We’ll use sklearn’s TFIDF vectorizer to eyeball some words important in the spam messages, and pick one to plug into our formula.
from sklearn.feature_extraction.text import TfidfVectorizervectorizer_spam = TfidfVectorizer(stop_words='english', max_features=30)vectorizer_spam.fit(spam_df['sms'])vectorizer_spam.vocabulary_
We need to pick a word to use in our formula so I’m going to choose the word “win”, though it would be interesting to try this for other words as well.
Now we need to calculate the different parts of our formula.
P(W|S) = probability of the word “win” being in a spam messageP(S) = probability of a spam message overallP(W) = probability of the word “win” in a message overall
Set our word.
word = 'win'
Calculate P(W|S).
word = 'win'spam_count = 0spam_with_word_count = 0for idx,row in spam_df.iterrows():    spam_count += 1        if word in row.sms:        spam_with_word_count += 1probability_of_word_given_spam = spam_count / spam_with_word_countprint(probability_of_word_given_spam)#=> 10.0
Calculate P(S).
probability_of_spam = len(spam_df) / (len(sample_df))print(probability_of_spam)#=> 0.12921751615218952
Calculate P(W).
sms_count = 0word_in_sms_count = 0for idx,row in sample_df.iterrows():    sms_count += 1        if word in row.sms:        word_in_sms_count += 1probability_of_word = word_in_sms_count / sms_countprint(probability_of_word)#=> 0.022254127781765973
Now putting it all together.
(probability_of_word_given_spam * probability_of_spam) / probability_of_word#=> 58.064516129032256
Boom. What this tells us is that if an SMS contains the word “win”, there is a 58% probability that the message is spam.
In a production spam detection system, we would need to do the above calculation for every word in our corpus, then combine the probabilities.
We may also want to include other features like combinations of words, message length, punctuation, etc.
That would turn this into a really long post.
If you’re interested, there is a PDF article here, explaining a couple approaches to combining results of multiple words.
I hope this gave you some insight and practical experience using Bayes theorem, even if we just scratched the surface.