The Ironic Sophistication of Naive Bayes Classifiers
By: Jack Ross
Tags: Naive Bayes, Naive Bayes Classifier, Naive Bayes In Python, Python, Naive Bayes From Scratch

The Ironic Sophistication of Naive Bayes Classifiers

Filtering spam with Multinomial Naive Bayes (From Scratch)

In the first half of 2020 more than 50% of all email traffic on the planet was spam. Spammers typically receive 1 reply for every 12,500,000 emails sent which doesn’t sound like much until you realize more than 15 billion spam emails are being sent each and every day. Spam is costing businesses 20–200 billion dollars per year and that number is only expected to grow.

What can we do to save ourselves from spam???

Naive Bayes Classifiers

In probability theory and statistics, Bayes’ theorem (alternatively Bayes’s theorem, Bayes’s law or Bayes’s rule) describes the probability of an event, based on prior knowledge of conditions that might be related to the event.

For example, if the risk of developing health problems is known to increase with age, Bayes’s theorem allows the risk to an individual of a known age to be assessed more accurately than simply assuming that the individual is typical of the population as a whole.

Bayes Theorem Explained

A Naive Bayes Classifier is a probabilistic classifier that uses Bayes theorem with strong independence (naive) assumptions between features.

Probabilistic classifier: a classifier that is able to predict, given an observation of an input, a probability distribution over a set of classes, rather than only outputting the most likely class that the observation should belong to.

Independence: Two events are independent if the occurrence of one does not affect the probability of occurrence of the other (equivalently, does not affect the odds). That assumption of independence of features is what makes Naive Bayes naive! In real world, the independence assumption is often violated, but naive Bayes classiﬁers still tend to perform very well.

Stopping the Spam

For this example we’ll be using the SMSSpamCollection dataset consisting of 5,572 messages. 86.6% of which are real messages and 13.4% that is spam/scams. We are going to build a Naive Bayes Classifier to predict if a message is spam or legitimate.

Above we can see our dataset has a Class feature (column) and a Message feature. Our 2 classes are ham (a legitimate message) and spam. We’ll load and split the data set into a training set and a testing set so later we can test our classifier on data it’s never seen.

import pandas as pd # Import the spam data set

email_data = pd.read_csv('/email-dataset.csv', sep='\t', names=['Class', 'Message']) # training_emails = 70% of the data

# testing_emails = 30% of the data from sklearn.model_selection import train_test_split train_emails, test_emails = train_test_split(email_data, test_size=0.3, random_state=42)

How It Works

When we get a new message and we want to determine if it is spam or not, the classifier will check a spam equation and a ham equation with each word of the message where “w1” is the first word, and (w1,w2, …, wn) is the entire message. Whichever one of these equations is greater, then that is what the new message is classifier as.

Spam Probability Equation

Ham (Authentic) Probability Equation

To calculate P(wi|Spam) and P(wi|Ham) — the last portion of the equations above — we need to use separate equations:

To clarify the terms above:

Creating the Vocabulary

We need to create a vocabulary of words to feed the classifier. After iterating through the dataset and counting all instances of different words, we’re left with a vocabulary of 7,301 unique words. This is what our new dataframe will look like after this step: