Algorithms From Scratch: Naive Bayes Classifier | by Kurtis Pykes | Towards Data Science

The Naive Bayes classifier is an Eager Learning algorithm that belongs to a family of simple probabilistic classifiers based on Bayes’ Theorem.
Although Bayes Theorem — put simply, is a principled way of calculating a conditional probability without the joint probability — assumes each input is dependent upon all other variables, to use it as a classifier we remove this assumption and consider each variable to be independent of each other and refer to this simplification of Bayes Theorem for predictive modelling as the Naive Bayes classifier. In other words, Naive Bayes assumes that the presence of a predictor in a class is not related to the presence of any other predictor. This is a very strong assumption given that it is very unlikely the predictors do not interact in real-world data.
By the way, if you are unfamiliar with Eager Learning, Eager Learning refers to the learning method in which the system aims to construct a general, input-independent target function during the training of the system. On the contrary, Algorithms such as K-Nearest Neighbors , which is a lazy learner, wait until a query is made before any generalization is made beyond the training data.
Essentially, Naive Bayes (or Idiot Bayes) earned its name due to the calculations for each class being simplified to make their calculations tractable, however the classifier proven itself to be effective in many real-world scenarios, whether it is a binary classification or multi-class classification, despite its naive design and oversimplified assumptions.
For the full code used in this notebook...
github.com
As we stated earlier, if we are going to apply Bayes Theorem for a conditional probability classification model (Naive Bayes model) then we need to simplify the calculation.
Before going into how we can simplify the model I am going to give a brief introduction into Marginal Probability, Joint Probability and Conditional Probability:
Marginal Probability — The probability of an event irrespective of other random variables, for instance P(A) where this means the probability of A occurring.
Joint Probability — The probability of two or more simultaneous events, for example P(A and B) or P(A, B).
Conditional Probability — Probability of one or more events given the occurrence of another event, for example P(A|B) which could be stated as the Probability of A given B.
A cool thing about these probabilities is that we could use them to calculate one another. The joint probability can be calculated by using the conditional probability — this is known as the product rule; See Figure 1.
An interesting fact about the product rule is that it is symmetrical meaning that P(A, B) = P(B, A). On the other hand, the conditional probability is not symmetrical meaning that P(A|B) != P(B|A), however what we can do is, we can calculate the conditional probability using the joint probability — see Figure 2.
There is one problem with Figure 2; It’s often quite difficult to calculate the joint probability, so when we want to work out the conditional probability we use an alternate method. The alternate method we use is called Bayes Rule or Bayes Theorem and it’s done by using one conditional probability to workout the other — See Figure 3
Note: We may also decide to use the alternative approach to calculating the conditional probability when the reverse conditional probability is available or easier to calculate.
To name the terms in Bayes Theorem, we must take into consideration the context of where the equation is used — See Figure 4.
Hence, we can restate Bayes Theorem as...
To model this as a classification model we do this...
However, this expression is a cause of complexity in our calculation so to simplify it we remove the assumption of dependence and considering each variable independent we simplify our classifier.
Note: We drop the denominator (the probability of observing the data in this instance) as it is a constant for all calculations.
And there you have it... Well not quite. This calculation is performed for each of the class labels but we only want to know the class that is most probable for a given instance. Hence, we must find the label with the largest probability to be selected as the classification for the given instance; The name of this decision rule is called Maximum A Posteriori (MAP) — Now there you have the Naive Bayes Classifier.
Chunking the Algorithm
Segment the data by the class, and then compute the mean and variance of x in each class.Calculate Probability using Gaussian Probability Density FunctionGet Class ProbabilitiesGet Final Prediction
Segment the data by the class, and then compute the mean and variance of x in each class.
Calculate Probability using Gaussian Probability Density Function
Get Class Probabilities
Get Final Prediction
Implementation
We are going to be using the iris dataset and since the variables used in this Dataset are numeric, we will build a Gaussian Naive Bayes model.
Note: The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(Xi | y) (Source: Scikit-Learn Naive Bayes)
import numpy as np import pandas as pdfrom sklearn.datasets import load_irisfrom sklearn.naive_bayes import GaussianNBfrom sklearn.metrics import accuracy_scorefrom sklearn.model_selection import train_test_split# loading the datairis = load_iris()X, y = iris.data, iris.target# spliting data to train and testX_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state=1810)X_train.shape, y_train.shape, X_test.shape, y_test.shape((120, 4), (120,), (30, 4), (30,))# scikit learn implementation nb = GaussianNB()nb.fit(X_train, y_train)sklearn_preds = nb.predict(X_test)print(f"sklearn accuracy:{accuracy_score(y_test, sklearn_preds)}")print(f"predictions: {sklearn_preds}")sklearn accuracy:1.0predictions: [0 0 2 2 0 1 0 0 1 1 2 1 2 0 1 2 0 0 0 2 1 2 0 0 0 0 1 1 0 2]
Scikit-Learn implementation gives us a perfect accuracy score on inference, Let’s build our own model and see if we can match the Scikit-learn implementation.
I have built a utility function get_params just so that we can get some parameters for our training data.
def get_params(X_train, y_train):     """    Function to get the unique classes, number of classes and number of features in training data    """    num_examples, num_features = X_train.shape    num_classes = len(np.unique(y_train))    return num_examples, num_features, num_classes# testing utility functionnum_examples, num_features, num_classes = get_params(X_train, y_train)print(num_examples, num_features, num_classes)120 4 3
Our utility function is working great so we can proceed to step one of getting the statistics (specifically mean, variance and the priors) by the class.
def get_stats_by_class(X_train, y_train, num_examples=num_examples, num_classes=num_classes):     """    Get stats of dataset by the class    """    # dictionaries to store stats    class_mean = {}    class_var = {}     class_prior = {}         # loop through each class and get mean, variance and prior by class    for cls in range(num_classes):         X_cls = X_train[y_train == cls]        class_mean[str(cls)] = np.mean(X_cls, axis=0)        class_var[str(cls)] = np.var(X_cls, axis=0)        class_prior[str(cls)] = X_cls.shape[0] / num_examples    return class_mean, class_var, class_prior# output of function cm, var, cp = get_stats_by_class(X_train, y_train)cm, var, cp# output of function cm, var, cp = get_stats_by_class(X_train, y_train)print(f"mean: {cm}\n\nvariance: {var}\n\npriors: {cp}")mean: {'0': array([5.06111111, 3.48611111, 1.44722222, 0.25833333]), '1': array([5.90952381, 2.80714286, 4.25238095, 1.33809524]), '2': array([6.61904762, 2.97857143, 5.58571429, 2.02142857])}variance: {'0': array([0.12570988, 0.15564043, 0.0286034 , 0.01243056]), '1': array([0.26324263, 0.08542517, 0.24582766, 0.04045351]), '2': array([0.43678005, 0.10930272, 0.31884354, 0.0802551 ])}priors: {'0': 0.3, '1': 0.35, '2': 0.35}
We passed the num_classes and num_examples that we got from get_params to the function as they are required to separate the data by class and calculate the priors by class. Now that we have the enough information to work out the class probabilities — Well not quite, we are dealing with continuous data and a typical assumption is that the continuous values associated with each class are distributed according to Gaussian distribution (Source: Wikipedia). As a result we built a function to calculate the density function that’s going to help us calculate the probability from a Gaussian Distribution.
def gaussian_density_function(X, mean, std, num_examples=num_examples, num_features=num_features, eps=1e-6):     num_exambles, num_features = X_train.shape    const = -num_features/2 * np.log(2*np.pi) - 0.5 * np.sum(np.log(std + eps))    probs = 0.5 * np.sum(np.power(X - mean, 2)/(std + eps), 1)    return const - probsgaussian_density_function(X_train, cm[str(0)], var[str(0)])array([-4.34046349e+02, -1.59180054e+02, -1.61095055e+02,  9.25593725e-01,       -2.40503860e+02, -4.94829021e+02, -8.44007497e+01, -1.24647713e+02,       -2.85653665e+00, -5.72257925e+02, -3.88046018e+02, -2.24563508e+02,        2.14664687e+00, -6.59682718e+02, -1.42720100e+02, -4.38322421e+02,       -2.27259034e+02, -2.43243607e+02, -2.60192759e+02, -6.69113243e-01,       -2.12744190e+02, -1.96296373e+00,  5.27718947e-01, -8.37591818e+01,       -3.74910393e+02, -4.12550151e+02, -5.26784003e+02,  2.02972576e+00,       -7.15335962e+02, -4.20276820e+02,  1.96012133e+00, -3.00593481e+02,       -2.47461333e+02, -1.60575712e+02, -2.89201209e+02, -2.92885637e+02,       -3.13408398e+02, -3.58425796e+02, -3.91682377e+00,  1.39469746e+00,       -5.96494272e+02, -2.28962605e+02, -3.30798243e+02, -6.31249585e+02,       -2.13727911e+02, -3.30118570e+02, -1.67525014e+02, -1.76565131e+02,        9.43246044e-01,  1.79792264e+00, -5.80893842e+02, -4.89795508e+02,       -1.52006930e+02, -2.23865257e+02, -3.95841849e+00, -2.96494860e+02,       -9.76659579e+01, -3.45123893e+02, -2.61299515e+02,  7.51925529e-01,       -1.57383774e+02, -1.13127846e+02,  6.89240784e-02, -4.32253752e+02,       -2.25822704e+00, -1.95763452e+02, -2.54997829e-01, -1.66303411e+02,       -2.94088881e+02, -1.47028139e+02, -4.89549541e+02, -4.61090964e+02,        1.22387847e+00, -8.22913900e-02,  9.67128415e-01, -2.30042263e+02,       -2.90035079e+00, -2.36569499e+02,  1.42223431e+00,  9.35599166e-01,       -3.74718213e+02, -2.07417873e+02, -4.19130888e+02,  7.79051525e-01,        1.82103882e+00, -2.77364308e+02,  9.64732218e-01, -7.15058948e+01,       -2.82064236e+02, -1.89898997e+02,  9.79605922e-01, -6.24660543e+02,        1.70258877e+00, -3.17104964e-01, -4.23008651e+02, -1.32107552e+00,       -3.09809542e+02, -4.01988565e+02, -2.55855351e+02, -2.25652042e+02,        1.00821726e+00, -2.24154135e+02,  2.07961315e+00, -3.08858104e+02,       -4.95246865e+02, -4.74107852e+02, -5.24258175e+02, -5.26011925e+02,       -3.43520576e+02, -4.59462733e+02, -1.68243666e+02,  1.06990125e+00,        2.04670066e+00, -8.64641201e-01, -3.89431048e+02, -1.00629804e+02,        1.25321722e+00, -5.07813723e+02, -1.27546482e+02, -4.43687565e+02])
Here is a function to calculate the class probabilities...
def class_probabilities(X, class_mean, class_var, class_prior, num_classes=num_classes):    """    calculate the probability of each class given the data    """    num_examples = X.shape[0]    probs = np.zeros((num_examples, num_classes))for cls in range(num_classes):         prior = class_prior[str(cls)]        probs_cls = gaussian_density_function(X, class_mean[str(cls)], class_var[str(cls)])        probs[:, cls] = probs_cls + np.log(prior)    return probs
Now we need to make a prediction using MAP so let’s put all these steps together in a predict function and output the max probability class.
def predict(X_test, X_train, y_train):     num_examples, num_features, num_classes = get_params(X_test, y_train)    class_mean, class_std, class_prior = get_stats_by_class(X_train, y_train)    probs = class_probabilities(X_test, class_mean, class_std, class_prior)    return np.argmax(probs, 1)my_preds = predict(X_test, X_train, y_train)print(f"my predictions accuracy:{accuracy_score(y_test, my_preds)}")print(f"predictions: {my_preds}")my predictions accuracy:1.0predictions: [0 0 2 2 0 1 0 0 1 1 2 1 2 0 1 2 0 0 0 2 1 2 0 0 0 0 1 1 0 2]
As a sanity check...
sklearn_preds == my_predsarray([ True,  True,  True,  True,  True,  True,  True,  True,  True,        True,  True,  True,  True,  True,  True,  True,  True,  True,        True,  True,  True,  True,  True,  True,  True,  True,  True,        True,  True,  True])
and that’s the way the cookie crumbles!
Require a small amount of training data to estimate the necessary parameters
Extremely fast compared to sophisticated methods
Known to be a bad estimator (In the Scikit-Learn framework, the outputs from predict_proba aren’t taken too seriously.
The assumption of independent predictors doesn’t hold true (most of the time) in the real world
You’ve now learnt about Naive Bayes Classifiers and how to build one from scratch using Python. Yes, the algorithm has very over-simplified assumptions, but it is still very effective in many real world applications and is worth trying if you want very fast predictions.
Let’s continue the conversation on LinkedIn...