Introduction to Naive Bayes Classifier
By: Priyanka Meena
Tags: Data Science, Naive Bayes, Probability, Classification Algorithms, Supervised Learning

Photo by Jorge Franganillo on Unsplash

Naive Bayes is a term that is collectively used for classification algorithms that are based on Bayes Theorem. For uninitiated, classification algorithms are those algorithms that are used to categorize a new observation into predefined classes. For example, let’s assume that you are working as a data analyst with a major bank in London and you wish to predict based on historical data, if a customer will fraud on a bank loan or not.

You must be wondering there is “Bayes” in the name because the algorithm is based on Bayes Theorem but why “Naive”? Is it because the algorithm is “Naive” or “Dumb”? No !! the algorithm is not “dumb” but in fact it works better than some very complicated algorithms at times. The algorithm is “Naive” because it works on the general assumption that the presence of a particular feature in a class is independent or totally unrelated to the presence of some other feature in the same class. For example, a client can default on a bank loan, if he/she has a bad credit score, low applicant_income etc. Both these features independently contribute to the probability that candidate will default, that is, presence of one feature is not related to another.

Don’t be disheartened if some of the terms sound alien to you. The purpose of this series of articles is to explain machine learning algorithms in the simplest possible way. So that by the end of this series, you will be able to build your own machine learning models with great ease. So let’s proceed with this article on Naive Bayes Classifier.

Bayes Theorem !! What exactly is Bayes Theorem?

Bayes Theorem is a very popular mathematical formula that is used to determine the conditional probability of an event, based on prior knowledge of conditions that might be related to the event.

Wait! What exactly is Conditional probability?

It is the likelihood of an outcome occurring,given that another event has already occured. For example, two cards are drawn without replacement from a deck of 52 cards. What is the probability that the second card is an ace given that the first card drawn was also ace?

So, P(drawing the first ace) = Total no. of aces / Total no. of cards = 4/52

P(drawing the second ace) = 3/51 (this is because after the first draw we are left with only 3 aces in the deck and the total number of cards also reduces to 51). So, this is what is conditional probability all about. The second event is depend on the occurrence of first one.

Coming back to Bayes Theorem. It is mathematically given by the following formula :

Fig 1 : Bayes Theorem [Image by Author]

Now that we have a decent understanding of what Bayes Theorem is. Let’s go ahead and understand its use in classification problems.

NAIVE BAYES INTUITION

Problem : Try to predict if a candidate with credit score of 180 will default on a loan or not. Consider the following frequency table for calculating the likelihood of default.

Fig 2 : Frequency table [Image by Author]

We can follow the following steps to calculate the probabilities. Since the candidate has credit score of 180, let’s predict the label for (100–200).

Step 1: Calculate the prior probability for each class (Yes and No)

P(Yes) = 11/29 = 0.379

P(100–200) = 10/29 = 0.345

Step 2: Find Likelihood probability with each attribute for each class

P(100–200 | Yes) = 6/11 = 0.545

Step 3: Calculate posterior probability using Bayes Theorem

P(Yes|100–200) = P(100–200|Yes) * P(Yes)/P(100–200)

= 0.545*0.379/0.345 = 0.5989 =0.599

Step 4: Make prediction

Since, the probability of the candidate defaulting is more than 50%(it’s the assumed significance level and it can differ based on the use case), we can say that the candidate will default.

END TO END NAIVE BAYES CLASSIFIER

Having learned how a naive bayes classifier works, let’s try to build a classification model based on it using sklearn. Sklearn or scikit-learn is an open source machine learning library written in python.

For the purpose of this article, we will be using social_network_ads dataset. In this problem, we will try to predict whether a user have purchased a product by clicking on the advertisements shown to him/her on social media, based on age and estimated salary. So let’s get started.

Step 1 : Import basic libraries

Step 2: Import Data

Fig 3 : Independent and Dependent variable overview [Image by Author]

Step 3: Data Preprocessing

In the data preprocessing step, we have first split the dataset into training(80%) and testing sets(20%). Next, we have done some basic feature scaling using standard scalar. Standard scalar transforms the data in such a way that it has a mean value of 0 and standard deviation of 1.

Step 4 : Model Training

Step 5: Model Testing and Evaluation

Fig 4 : Confusion matrix [Image by Author]

A confusion matrix gives us an idea of how good is our model. It describes the performance of a classification model on a set of test data for which the

true values are known.Each row in a confusion matrix represents an actual class, while each column represents a predicted class.

Step 6 : Visualization

Fig 5 : Visualization of Naive Bayes classification model [Image by Author]

This step is not necessary. It has been put up just to give an idea of how data points have been classified by the model.

WRAP UP

In this post, we have a learned the mathematics behind Naive Bayes classifier and built a model using sklearn. In the coming articles, we will be learning about some other classification algorithms as a part of this series such as logistic regression, decision trees etc. So stay tuned.