Understanding Naïve Bayes algorithm
By: Vaibhav Jayaswal
Tags: Naive Bayes, Machine Learning, Classification, Bayes Theorem

Understanding Naïve Bayes algorithm

Image by Riho Kroll, Unsplash

Naïve Bayes is a classification algorithm that is a probabilistic classifier based on Bayes theorem. Before getting into the intricacies of Naïve Bayes, we first understand the Bayes theorem.

Bayes Theorem

Bayes theorem says that if event B has occurred, then we can find the probability of event A given B.

Mathematically, Bayes theorem is represented as

Where P(B) != 0 (!= refers to not equal)

P(A|B) — the probability of event A given B (called posterior)

P(B|A) — the probability of event B given A (called likelihood)

P(A) — the probability of event A (called prior)

P(B) — the probability of event B (called evidence)

For example, if we have a query point X (d-dimensional Boolean vector), we have to predict which class (Y) it belongs to out of two. How to calculate the probability of Y given X?

Problem in the above representation

Think of how many combinations can exist if we have the above formula. X being a d-dimensional Boolean vector, 2^d possible combinations for vector X, and 2 for output (2 class classification). The total combinations are 2^(d+1) for P(X1, X2,……, Xd|Y), which is massive and not effective in real-world scenarios.

Image by Andrea Piacquadio, Unsplash

Naïve Bayes handles that problem by making an assumption.

Naïve Bayes

Assumption — Each feature in the input vector is conditionally independent of other features.

Mathematically.

It states that A is conditionally independent of B given C.

How can this help?

The numerator, i.e., P(X1, X2,…., Xd|Y)*P(Y), is equivalent to the joint probability model and can be expressed as P(X1, X2,…., Xd, Y). Then we can unfold it using conditional probability.

On applying the assumption of conditional independence, it can be written as

Therefore, the probability formula using conditional independence is

The advantage is that we only have to find (2d+2) values when assuming conditional independence, which is significantly less as compared to 2^(d+2) (2^(d+1) for P(X1, X2,……, Xd|Y) and 2 for P(Y)).

If we have two classes, we can get the probability of the class Y for class 1 P(Y=1|X) and class 0 P(Y=0|X) given a query point (X). The class assigned to the query point is decided based on the MAP decision rule (maximum a posterior), i.e., the class with a higher probability given X (P(Y|X)).

Example

Given the features such as outlook, temperature, and windy, predict whether the conditions are appropriate for playing tennis or not.

X = [outlook, temperature, windy]

Y = [Yes, No]

Image by Author

Since Naïve Bayes follows conditional independence, we will calculate P(Xi|Yj) for all i in X and j in Y.

We will calculate the likelihood tables.

Table 1

Outlook P(Outlook|class)- where outlook can be overcast, rain and sunny and class can be yes and no.

Table 2

Temperature P(Temperature|class)- where temperature can be hot, mild, and cool and class can be yes and no.

Table 3

Windy P(Windy |class) - where Windy can be strong and weak and class can be yes and no.

Priors are

P(class=Yes) = 9/14

P(class=No) = 5/14

So if we have a query point, how to calculate which class it belongs to?

Given a query point X’ =[sunny, cool, strong]

We calculate P(class=Yes|X’) and P(class=No|X’)

P(X’) is a constant term which is equal to

From the likelihood table, we get the values required in the formula

Since the probability of class ‘No’ given X’ is higher than class ‘Yes’ given X’, the query point belongs to class ‘No’.

Advantages

The naïve Bayes algorithm can also perform multiclass classification by comparing all the classes’ probability given a query point. Naïve Bayes algorithm is efficient on large datasets since the time, and space complexity is less.

Run time complexity is O(d*c) where d is the query vector’s dimension, and c is the total classes. Space complexity is also O(d*c) since we are storing only the likelihood of each feature with respect to classes, i.e., P(Xd|Yc) where d is the feature and c is the class. Therefore, the total combination is (d*c). Naïve Bayes works reasonably well in text classification problems such as spam filtering and reviews classification.

There are different types of NB, such as Gaussian Naïve Bayes, which works well with continuous data, Multinomial Naïve Bayes for text classification by using the frequency of each word in a document. Another type is Bernoulli Naïve Bayes, which only works if the features take two values (0 and 1) similar to that in the binary Bag of Words model.

Conclusion

Naïve Bayes is a classification algorithm that works on the Bayes theorem with a conditional independence assumption. It works well in real-world applications that require low latency because of the low time and space complexity.

Thanks for reading!