Understanding Bayes’ Theorem
By: Tony Yiu
Tags: Machine Learning, Artificial Intelligence, Data Science, Statistics, Education

The Intuition Behind the Formula

Remember how we said that Bayes’ Theorem is a framework for updating our beliefs? So where do our beliefs come in? They come in through the prior, P(A), which in our example is P(Offer) — this is our prior belief about how likely it is to receive an offer. In our example, you can think of the prior as our belief of the likelihood that you will receive an offer at the exact moment that you exit the interview room.

Now, new information has come in — 3 days have gone by and the company has yet to call you. So we use the other parts of the equation to adjust our prior for the new event that has occurred.

Let’s examine P(B|A), which is P(NoCall|Offer) in our example. When you first learn Bayes’ Theorem, it’s natural to wonder what the point of the P(B|A) term is. If I don’t know P(A|B), then how am I supposed to magically know what P(B|A) is? This reminds me of something that Charles Munger once said:

“Invert, always invert!” — Charles Munger

What he meant is that when trying to solve a challenging problem, it’s easier to turn the problem around on its head and look at it backwards — which is exactly what Bayes’ Theorem is doing. Let’s reframe Bayes’ Theorem into statistical terms to make it more interpretable (I first read about this here):

Bayes’ Theorem reframed so that it is more intuitive

To me, this is a much more intuitive way of thinking about the formula. We have a hypothesis (that we got the job), a prior, and observed some evidence (no phone call for 3 days). Now we want to know the probability that our hypothesis is true given the evidence. As we discussed above, we already have our prior of 20%.

Time to invert! We use P(Evidence|Hypothesis) to flip the problem around by asking, “What is the probability of observing this evidence in a world where our hypothesis is true?” So in our example, we want to know how likely it is to go 3 days without a phone call in a world where the company has definitely decided to make you an offer. In my annotated image of the formula above, I call P(Evidence|Hypothesis) the scaler because that’s exactly what it does. When we multiply it against the prior, the scaler scales the prior up or down depending on whether the evidence helps or hurts our hypothesis — in our case, the scaler reduces the prior because more days going by without a phone call would be an increasingly bad sign. 3 days of radio silence is already not good (it reduces our prior by 60%), but 20 days of silence would completely destroy any hope we had of getting the job. So the more our evidence accumulates (more days without a phone call) the more the scaler reduces our prior. The scaler is the mechanism that Bayes’ Theorem utilizes to adjust our prior beliefs.

EDIT: One thing I struggled with somewhat in the original version of this post is articulating why P(Evidence|Hypothesis) is easier to estimate than P(Hypothesis|Evidence). The reason for this is thatP(Evidence|Hypothesis) is a much more constrained way of thinking about the world — by narrowing the scope, we simplify our problem. An easy way to see this is with the popular fire and smoke example where fire is our hypothesis and observing smoke is the evidence. P(fire|smoke) is harder to estimate because any number of things can cause smoke — exhaust from cars, a factory, someone grilling burgers over a charcoal flame. P(smoke|fire) is much easier to estimate — in a world where there is a fire, there will almost certainly be smoke.

The value of the scaler decreases as more days pass with no call — the lower the scaler, the more it reduces the prior as they are multiplied together

The last part of our formula, P(B), a.k.a. P(Evidence) is the normalizer. Like the name implies, its purpose is to normalize the product of the prior and the scaler. If we didn’t divide by the normalizer, we would have the following equation:

Notice that the product of prior and scaler is equal to a joint probability. And because one of the terms in it is P(Evidence), the joint probability would be impacted by the rarity of our evidence.

This is problematic because the joint probability is a value that considers all states of the world. But we don’t care about all states — we only care about the ones where the evidence has occurred. In other words, we are living in a world where the evidence has already occurred and the abundance or scarcity of our evidence is no longer relevant (so we don’t want it to affect our calculation at all). Dividing the product of prior and scaler by P(Evidence) changes it from a joint probability to a conditional one — a conditional probability is one that only considers the states of the world where the evidence has occurred, which is what we desire.

EDIT: Another way to think about why we divide the scaler by the normalizer is that they answer two different and important questions — and the ratio of them combines the information in a useful way. Let’s use an example from my new post on naive Bayes. Say we are trying to figure out whether an observed animal is a cat based on a single feature agility. All we know is that the animal in question is agile.