Bayes’ Theorem, Simply Explained. If ignorance is bliss, knowing how to... | by Riccardo Di Sipio | Towards Data Science

Bayes’ Theorem formalizes how to calculate the probability of an event to occur given our incomplete knowledge about other factors affecting its realization, and our pre-existing belief. For example, is it going to rain today if the sky is cloudy?
Let’s start with some basic definitions. The probability of an event A to occur is a number between 0 and 1 (or 0% and 100%) and is represented mathematically by some function P(A) called marginal probability, i.e. irrespective of other events. If we are considering two events A and B at the same time, then the probability is a 2-dimensional function P(A,B) which is called joint probability. Notably, it’s symmetrical, meaning that P(A,B) = P(B,A). If the two events are completely uncorrelated, then it factorizes into P(A,B) = P(A)*P(B).
The next thing we may want to know is what’s the probability of event A to occur knowing that B already happened. This is called conditional probability, described by a function P(A|B). There is a well-known principle in statistics called product rule which tells you how to derive the conditional probability P(A|B) from the joint probability P(A,B) and the marginal probability of the conditioning event i.e. P(B). It’s actually pretty simple:
P(A|B) = P(A,B) / P(B)
The division by P(B) ensures that the probability is between 0 and 1. Notably, the conditional probability is not symmetrical, meaning that in general P(A|B) ≠ P(B|A). For example, the chance of rain with an overcast sky is arguably smaller (e.g. 30%) than the chance of having an overcast sky while its raining (e.g. 99%).
These relationships can be inverted easily:
P(A,B) = P(A|B) * P(B)P(B,A) = P(B|A) * P(A)
which basically shows that having the joint probability is the key to hold complete information about a system.
The catch is, in most of the cases our knowledge of a certain process is incomplete, which means we do not have access to the joint probability P(A,B)! On the other hand, we often have some knowledge about event A happening under the assumption that event B is true, and we probably have some pre-existing ideas about how likely event A is in general. Then, a theorem devised by revered Thomas Bayes comes to rescue to tells us how to “switch” between the two conditional probabilities, i.e.:
P(B|A) = P(A|B) * P(B) / P(A)
This is easily derived by realizing that:
P(A|B) * P(B) = P(A,B) P(B|A) * P(A) = P(B,A) = P(A,B)=>P(A|B) * P(B) = P(B|A) * P(A)=>P(B|A) = P(A|B) * P(B) / P(A)
In the Bayesian statistics lingo, P(B|A) is the posterior probability (what we want to calculate), P(A|B) is called the likelihood, P(B) is called prior probability, and the normalization factor P(A) is the marginal likelihood or model evidence. In plain English, the likelihood encodes the probability that event A can occur under some set of assumptions B, and the prior is our belief that B is actually possible. To stretch it a little bit: what is the probability that God exists given that we have observed what looked like a miracle, and our belief that miracles can happen at all irrespective of the existence of a God? Maybe that’s what Thomas Bayes was mulling over.
To summarize:
Joint probability: P(A and B) = P(A, B) = P(B, A)
Conditional probability: P(A given B)= P(A|B)
Product rule: P(A|B) = P(A, B) / P(B)
P(A|B) * P(B) = P(A, B) = P(B, A) = P(B|A) * P(A)
P(B|A)= P(A|B)*P(B) / P(A) (Bayes’ Theorem)
If you are reading this post, it’s probably because you heard about the Naive Bayes Classifier and its success at rejecting unwanted emails. How does it work, and why is it “naïve”?
First of all, we are talking about a classifier, which is a statistical model that separates instances into different (often mutually-exclusive) categories based on the value of some features vector X. In the example mentioned above, the features may be whether it’s raining or not, and the categories whether you should bring an umbrella or not. A full-fledged application of Bayes’ Theorem would require quite some computations, for example that of the normalization factor P(X) which often implies running a Markov Chain Monte Carlo integration over the multi-dimensional feature space. However, in most applications, we only want to know the category, not the (normalized) probability. This way, the theorem can be simplified to:
P(umbrella | X ) ∝ P(X | umbrella) * P(umbrella)
The other big problem is the correlation among the features X. In real life, there are correlations between the level of humidity, temperature and time of the day, and the classifier should take this fact into account during the optimization step. For example, an increase in humidity may be correlated with an increase in temperature. However, in practical terms, it’s often the case that the input features are treated or engineered in such a way that their correlation is in fact very small or negligible. If that’s the case, then the likelihood (remember it’s a conditional probability!) factorizes into:
P(X|umbrella) = P(rainy|umbrella) * P(cloudy|umbrella) * P(humid|umbrella) * P(hour|umbrella)
Thus, for a given instance of the features vector X = (X_1, X_2, X_3, X_4), the algorithm just has to find the maximum of P(rainy=X_1|umbrella), P(cloudy=X_2|umbrella), etc.. separately, and then multiply the factors together:
umbrella = argmax_j P(umbrella_j) * Prod_i[ P(X_i|umbrella_j) ] = max{P(Y) * P(X_1|Y) * P(X_2|Y) * P(X_3|Y) * P(X_4|Y),P(N) * P(X_1|N) * P(X_2|N) * P(X_3|N) * P(X_4|N)}
In many cases, handling a product of perhaps a hundred features can cause numerical instabilities or memory overflows. It is customary then to take the negative log of the product, and instead minimize the function:
-logP(X|umbrella) = -logP(umbrella) - log(P(X_1|umbrella) - log(P(X_2|umbrella) - log(P(X_3|umbrella) - log(P(X_4|umbrella)
To conclude, Bayes’ Theorem is a formal way to calculate the probability of a given event A based on incomplete knowledge about its relationship with other events B_1, B_2, ... and our prior beliefs. Its importance in statistics cannot be overstated, and its applications are ubiquitous. We discussed here the Naive Bayes Classifier, a popular approach implemented for example in the scikit-learn library. I also discussed how to train Neural Networks with Bayesian optimization in a previous post.