Why is Naive Bayes’ theorem so Naive?
By: Chayan Kathuria
Tags: Probability, Machine Learning, Data Science, Bayesian Statistics, Bayes Theorem

Intuition

So consider the previous example of a box with 3 red and 2 black balls. The marginal probability of picking up a red ball on the first go will be 2/5. Let this be P(A). Now, from the remaining 3 red and 1 black ball, the probability of drawing another black ball will be 1/4 which is P(B|A). Now this is the conditional probability of drawing a black ball given a black ball has already been drawn out during the first go, which was event A.

Now if we multiply both these probabilities, we will get 1/10 which is the joint probability P(A,B).

Multiplication Rule

Plugging in the values in the above equation for P(A,B) and P(B), we will get P(B|A) = 1/4. So 1/4 is the conditional probability of event B when A has already occurred (dependent events).

Similarly, we can also define P(A|B) as:

Multiplication Rule

Now we very well know that P(A,B) = P(B,A). Therefore, equating both the equations we get:

Bayes’ theorem

Here, P(B) is the prior probability, P(A|B) is the likelihood, P(A) is the marginal probability and P(B|A) is the posterior probability.

Naive Bayes’ theorem

Intuition

Bayes’ theorem which we just discussed above is so ridiculously simple and can be used in classification tasks, be it binary or multi class classification.

Consider we have a classification Machine Learning problem at hand. Suppose we have 5 features X1, X2, X3, X4 and X5 and the target variable is Y. Now we need to fit our data into this Bayes’ theorem such that it is able to predict the probability of class Y, given a certain set of 5 data points (See the application of Bayes’ rule here).

Assuming all the features are independent

We get the above equation. This can be presented as:

Here the Pi symbol is just doing the summation of the products of the likelihood probability. Now if you look it closely, the denominator or the marginal probability will be constant for all the instances. Hence we can define the proportionality for the above equation as:

Now this will give us the probabilities for both the classes. One will be higher (which we will consider as the predicted class) and other one will be lower. Hence we will take the argmax() of it to obtain that value.

Example

Now consider following simple dataset where we have 2 tables — Outlook and Temperature. Here Outlook has 3 possibilities viz., Sunny, Overcast and Rainy and the result in Yes/No as to whether the man will play tennis or not.

Outlook

Similarly the second table consists of data regarding temperature and its effect on the outcome Yes/No.

Temperature

And the total probabilities of Yes and No will look like this:

Now the problem at hand is that we need to find whether the man will play or not if the weather is Sunny and the temperature is Hot. Or in terms of probability we need to find P(Yes|Today) where Today is (Sunny,Hot). Here weather and temperature are nothing but the 2 features of our dataset. So the equation becomes:

Solving the above for both P(Yes|Today) and P(No|Today) by plugging the values from the tables above:

P(Sunny) and P(Hot) won’t be considered at is constant

So looking at the probabilities it is evident that P(No|Today) is higher, so the prediction for this instance will be ‘No’. But, these are not the class probabilities and you might have noticed that they don’t add up to 1. So we need to normalize the above conditional probabilities to get the class probabilities:

And to get the class probability for No, we can simply subtract this probability from 1. Hence probability of No will be 1–0.27 = 0.73.

Therefore the algorithm will predict the class as ‘No’.