How to Consume News More Intelligently Using Bayes’ theorem
By: Ray Heberer
Tags: Data, Rationality, News, Mathematics, Statistics

How to Consume News More Intelligently Using Bayes’ theorem

Base rates, marginal probabilities, sensitivity, and specificity

Photo by Markus Spiske on Unsplash

When it comes to updating beliefs and making decisions under uncertainty, Bayes’ theorem is just about the best tool available. And yet it is so often relegated to academic textbooks and machine learning applications when it should be bringing us value in our daily lives.

This article is for those of us who are familiar with Bayes’ theorem, but might have trouble remembering which terms go where and don’t immediately think of it when reading headlines like “red wine reduces dementia risk.”

First, I’ll review Bayes’ theorem, talk about why I think it is hard to remember when it really shouldn’t be, and formulate it in terms of sensitivity and specificity (which I’ll define). Next, I’ll discuss a few ways to use the formula to appropriately react to provocative claims that often appear in the news. Finally, I’ll leave you with some caveats about using Bayes’ theorem in an everyday fashion, and other final thoughts.

The key takeaways are as follows. I will define all the terms that appear:

Bayes’ theorem tells us how to revise a probability estimate of some outcome, given some new condition. In everyday life, it can be useful in simply updating our beliefs when presented with more evidence, or in determining whether an outcome (e.g. cancer) is significantly more likely given some condition (e.g. smoking). You need three pieces of information to properly update your beliefs about an outcome: the base rate of the outcome, and the sensitivity and specificity of the evidence. News often only gives you the sensitivity. In the absence of all three pieces of information, try to fill in the blanks yourself and be wary of situations with extreme (high/low) base rates, or a high marginal probability of the evidence.

Bayes’ Theorem in terms of Sensitivity and Specificity

Bayes’ Theorem tells us how to calculate a posterior probability of an outcome, given new evidence. This is expressed as a conditional probability of the form P(A|B).

When the formula is expressed as a jumble of letters, it’s easy to get them mixed up and forget what goes in the numerator and what goes in the denominator. For this reason, I like to write it out in terms of outcomes and conditions.

Mathematically, there isn’t really a reason to call one event an “outcome” and another a “condition.” But in real life, there are things that we would like to know but can’t measure directly — for example, the probability that you have cancer. This would be an outcome. Pieces of evidence like the results of a medical diagnostic are conditions used to estimate such outcome probabilities.

P(outcome) is what’s known as the prior probability or base rate of the thing we’re interested in. It makes sense that it would be proportional to our updated probability since otherwise, we would be flipping our beliefs every time we encountered new evidence.

P(condition) is called the marginal probability of the condition, and it can be a little strange to think about as one entity. It is the total probability of observing the condition or evidence that we did, in both the worlds where the outcome was true and where it was not.

Now, there are two more terms I’d like to introduce: sensitivity and specificity. Let’s say you read that 70% of medical students play the piano. That means that playing the piano as evidence of being a medical student has a sensitivity of 70%.

But surely being a piano player or not isn’t a strong indicator of whether or not someone is a medical student. One reason for this would be its low specificity. That is to say, a lot of people who aren’t medical students also play the piano.

In order for a condition to strongly influence the probability of an outcome, it must have both high sensitivity and high specificity. Furthermore, in order to come up with a posterior probability estimate at all, we must have an idea of the base rate of the outcome in addition to the sensitivity and specificity of the evidence. Below is Bayes’ formula rewritten in a way that is consistent with these intuitions:

I’ve found that news articles often don’t provide all the details necessary to come up with a fully data-informed belief. The amount by which Bayes’ theorem tells us to adjust our beliefs can often be counter-intuitive, especially when dealing with rare events. In the next section, I’ll describe how to avoid overreacting to statements like “90% of COVID-19 cases occur in individuals who wear shoes with shoelaces.”

Scenarios to Watch Out For

Again, three pieces of information are needed to compute a posterior probability of an outcome: the base rate of the outcome, the sensitivity of the condition, and the specificity of the condition. In particular, if the sensitivity or specificity are nothing special (~0.5), then that’s a sign that the posterior won’t be too different from the base rate. In such situations, assuming your current behaviors are based on your beliefs, you probably won’t need to adjust what you’re doing to account for the new data.

But what should you do when only one or two pieces are provided by the news? Well, your options are:

Admit that you can’t meaningfully incorporate the new data, and ignore it. Estimate the missing pieces.

While there are some situations where you really can’t estimate the missing pieces, it’s dangerous to lean on this option too much. I find that often sensitivity is the piece provided in the news, while base rate and marginal probability (and thus specificity) can be looked up without too much difficulty.

In such scenarios, I’ve learned to be wary whenever my estimates surface low base rates, high marginal probabilities, or high base rates. Let’s examine each situation in turn, using fictional examples.

Low Base Rate

“90% of mothers with children having genius-level IQ report eating shellfish during pregnancy.”

In situations such as the example above, the outcome we are interested in (having a child with genius-level IQ) is quite rare to begin with. This means that unless the condition we are testing for is razor sharp in terms of sensitivity and specificity (which it probably isn’t if it wasn’t directly reported in a scientific journal), our posterior probability will probably also be quite low.

High Marginal Probability

“Nearly every American university student seeking depression counselling owned a smartphone.”

Like being an owner of shoes with shoelaces, owning a smartphone is now quite common, depression counselling aside. In other words, the marginal probability of owning a smartphone is high. This can be a warning sign that the specificity of the condition is low.

In other words, we can assume that nearly every university student who did not seek depression counselling also owned a smartphone, meaning that owning a smartphone on its own is not a strong indicator that a student will seek counselling.

High Base Rate

“95% of people who have been on an airplane die before the age of 100.”

Before you read something like this and decide your flying days are over, consider that dying before the age of 100 is already quite common. There’s a chance that the posterior probability of dying before that 100th revolution might be lower than the base rate. And in any case, the posterior probably won’t be different enough to warrant a lifestyle change.

Caveats and Concluding Thoughts

I hope that I’ve managed to provide another angle to view Bayes’ theorem from which will make it more intuitive and easier to use in everyday life. But with great power comes great responsibility, so be mindful of the following caveats.

Even if you determined that some new information didn’t adjust your beliefs enough to change your behavior, that’s not the same thing as having it not adjust your beliefs at all. If more and more evidence towards the same thing builds up, eventually that should change your overall views. It will do you no good to forget each piece of evidence and revert to your prior because you’ve deemed it as insufficient.

And again, be wary of rejecting news outright because it doesn’t explicitly provide all the ingredients needed to run Bayes’ rule. This runs the risk of leading to motivated skepticism, if you only apply such a filter on news you dislike.

While in my examples above I made sensitivity claims directly, headlines will often make claims about the outcome directly (e.g. “smartphones increase the chance of depression”) but then only report the sensitivity of the evidence in the body. Look past such headlines and find the numbers.

With these points in mind, you should be equipped to react to the information you encounter in a more measured and data-minded fashion. Cheers!

P.S. Anecdotal evidence is still worthless within this framework.