Implementing Naive Bayes Algorithm from Scratch — Python.
By: Iuliia Stanina
Tags: Lambda School, Naive Bayes Classifier, Computer Science, Data Science, Data Visualization

Implementing Naive Bayes Algorithm from Scratch — Python.

In this article, I will show a basic implementation of Naive Bayes Classifier and how it works under the hood.

Image taken from www.i2tutorials.com

Introduction

Naive Bayes Classifier is a very popular supervised machine learning algorithm based on Bayes’ theorem. It is simple but very powerful algorithm which works well with large datasets and sparse matrices, like pre-processed text data which creates thousands of vectors depending on the number of words in a dictionary. It works really well with text data projects like sentiment data analysis, performs good with document categorization projects, and also it is great in predicting categorical data in projects such as email spam classification.

It is used to solve many different problem statements, and it is quite fast in training a model since Naive Bayes classifier completely works on probability, so the conversion happens quickly.

Understanding Math and Statistics behind

Bayes’ theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event.

First, let’s take a formula of conditional probability, and try to derive Bayes Theorem:

P(B|A) = P(A∩B)/P(B),

where probability of B given A, meaning what is the probability of event B when event A is already taken place, equals probability of A intersection B (meaning the probability of both A and B events taken place) divided by probability of B.

or P(A|B) = P(B∩A)/P(A),

where probability of A given B, meaning what is the probability of event A when event B is already taken place, equals probability of B intersection A (meaning the probability of both B and A events taken place) divided by probability of A.

Let’s take a closer look, we see that P(A∩B) and P(B∩A) are basically the same, so we can write them as P(A∩B) = P(B∩A). Since they are the same, we can get two formulas and move denominator to the left of the equation:

P(A∩B) = P(A|B) * P(B), and P(B∩A) = P(B|A) * P(A) and equate them:

P(A|B) * P(B) = P(B|A) * P(A).

So, when we want to find probability of A given B we can write our equation this way:

P(A|B) = P(B|A) * P(A) / P(B), and this is the equation of Bayes Theorem.

Applying Bayes Theorem Equation in Algorithm

Let’s break down our equation and understand how it works:

P(A|B) is the posterior probability of class (target) given predictor (attribute).

is the posterior probability of class (target) given predictor (attribute). P(B) is the prior probability of class.

is the prior probability of class. P(B|A) is the likelihood which is the probability of predictor given class.

is the likelihood which is the probability of predictor given class. P(A) is the prior probability of predictor.

In order to calculate posterior probability, first we need to calculate frequency for each attribute against the target. Then, transform the frequency to likelihood values and finally use the Naive Bayesian equation to calculate the posterior probability for each class. The class with the highest posterior probability is the outcome of prediction.

For my example I used two datasets:

Iris dataset to predict type of flowers Spambase dataset to detect if emails contain spam

Since both datasets have continuous attributes, I chose a Gaussian distribution to estimate the parameters for the likelihood probabilities in Naive Bayes classifiers. The distribution is characterized by two parameters, its mean and variance. The formula for Gaussian Density function, derived from Wikipedia, looks like this: (1/√2pi*σ) * exp((-1/2)*((x-μ)²)/(2*σ²)), where μ is mean, σ² is variance, σ is square root of variance (standard deviation).

First we need to calculate mean and variance for each column and convert it to numPy array for future calculations:

calculate mean and variance for each column

Next, let’s convert Gaussian density function to code:

calculate probability from gaussian density function

The last step is to calculate prior and posterior probabilities:

Making Predictions

Finally, all the helper methods are now ready to use them in fit and predict methods:

fit and predict methods

After training my model on the iris flowers classification data, I got really good accuracy score of 92%.

True vs Predicted in iris flowers classification

I compared it to scikit-learn Gaussian Bayes Classifier and got the same results, which is very impressive, since my algorithm is just barebone and needs a lot of improvement.

For the email spam classification problem my model performed only 78% of accuracy, however it is not so far from scikit-learn Gaussian Bayes Classifier which had an accuracy of 81.8%.

True values vs Predicted in email spam classification

Conclusion

My Naive Bayes Classifier algorithm implementation is far from ideal , it requires many improvements and modifications to make a better predictions especially for text data, however it’s still performs pretty good in comparison to the sklearn library’s one.

Note:

This project was done for Lambda School as a part of the Computer Science curriculum.

Github link for code implementation: Naive Bayes Classifier from Scratch