An Introduction to Simpson’s Paradox
By: David Ramsay
Tags: Bayesian Statistics, Judea Pearl, Simpsons Paradox, Data Science, Causality

Simpson’s paradox is a fascinating phenomena that illustrates the importance of causality in reasoning. If you haven’t already read it, you should check out The Book of Why — it’s one of the most important books of this decade.

I think Simpson’s paradox is a great introduction to the value of causal reasoning. In this post I’m going to walk through a few examples to elucidate how it arises, and how we can start to formalize our thinking around it.

Batting Averages

A common example of Simpson’s Paradox comes from Ken Ross. He showed that when looking at Derek Jeter’s and David Justice’s batting averages, a curious contradiction arises:

David Justice has a better batting average every year, but Derek Jeter has a better batting average overall!

Derek Jeter has a worse batting average every season, but somehow has a better batting average overall!

If we stare at this for a moment, the reason becomes clear as a few things stand out. In the first year, Derek and David both had low scores; in the second year they both had high scores. Most importantly, Derek’s low score was only from a few at bats, and thus doesn’t factor into his overall average very strongly. David, on the other hand, is exactly the opposite — his lower score makes up the vast majority of his attempts at bat. Even though David did slightly better in both years without weighting, when we combine this imbalance of at bats with their similarly large difference in score, we get this result. Simpson’s paradox shows us that a properly weighted average for an overall statistic can contradict the trends we see in every subgroup.

Another simple example — this time graphical — is below. Let’s say very young people (who tend to be shorter than adults) practice a lot of basketball and are pretty good. Let’s say older folks (who tend to be taller) quit practicing and generally are actually pretty bad at shooting hoops.

Now let’s say we measure their performance shooting hoops, with the hopes of finally proving that taller people are better at basketball. And what we get looks like this:

A graphical example of Simpson’s paradox. Taller kids and taller adults are both better at basketball, but in general being tall makes you worse!

Another conundrum! It appears that — for each age group — being taller means you are a better basketball player. Overall, though, being taller makes you worse!

We all know that being tall makes you better, and the right way to interpret this result is separated by age instead of using the overall population. But why and how do we know that? And how do we generalize that rule of thumb to trickier situations?

Causal Reasoning

Here’s a more difficult example, which mirrors a real life medical study. Imagine you’re sick, and you’re offered a drug that had the following results for previous patients:

Uh oh. The drug is good for men, good for women, and bad for people.

You read it correctly. This drug improves your odds of recovering if you’re a man; it also improves your odds of recovering if you’re a woman. But if you’re a person, it’s more likely to harm you. Do you take this drug?

Thankfully, the above statement is logically incoherent and false. But it’s easy to read the data and see it that way. How can we address this problem?

The Answer

We *know* that there are three things at work here to give us Simpson’s paradox. There’s (1) the gender of the patient, (2) how effective the drug is, and (3) how likely a patient is to be in a treatment or non-treatment group. Remember, it’s the imbalance of the groups that makes it possible for the overall average to be weighted differently then the trend in each subgroup.

The question then becomes, where are the causal relationships? It’s safe to say that your gender isn’t going to change based on your treatment or recovery. On the other hand, gender does seem to influence how likely you are to choose the treatment — women chose it 75% of the time, while only 12% of the men did.

Given we’re examining the data in this way, we clearly believe that gender might influence recovery. Finally (and obviously), treatment will affect your chance of recovery and not the other way around. We can draw a nice little causal diagram to summarize all of our causal knowledge.

A simple causal diagram of what’s going on. Gender alters the likelihood of accepting treatment. We also expect the likelihood of recovery to be a function of both receiving treatment AND gender.

Where do things go wrong? Hopefully it’s clear that the ‘gender-influenced treatment selection’ part of the diagram — that arrow linking gender to treatment — is what is causing our paradox.

We don’t want that information to alter our decision. How likely someone is to choose the treatment doesn’t tell us anything about how effective the treatment ultimately is for them. To get an accurate account for how effective this drug is for people as a whole, we must eliminate that link. In other words, we should just average the percentages of each group, without weighting them by size. This removes the effect of uneven treatment selection by the two genders.

Simpson’s paradox is resolved!

Does It Generalize?

In the case above, it made sense to trust the sub-groupings and not the overall average. In other cases, it can be the opposite.

Take an example (this one from Judea Pearl’s book) where we look at a drug designed for heart attack prevention, and we assess its effect on low-blood pressure and high-blood pressure individuals. We see a similar issue — the drug decreases heart attacks in people with low- and high- blood pressure, but increases heart attacks overall! What do we do? In this case, when we draw a causal diagram, it might look something like this:

Causal Diagram for our second example. In this case, the blood pressure is a mediating variable in the path from treatment to heart attack, and thus we should trust the overall statistic.

This looks very similar to our first example, but it’s not. Look closely. The locations of our subgroups and the treatment have switched.

In the first case, our subgroup (gender) affected who receives treatment. In the second, our subgroup (blood pressure) is affected by who receives treatment.

In this case, both the risk of heart attack and the patient’s blood pressure are causally downstream from treatment, so we should not subdivide by blood pressure. Blood pressure is a mechanism by which the treatment is working — it’s a mediating variable along the causal path from treatment to heart attack risk.

Contrast that to the prior case, where the relationship between treatment and recovery was confounded by gender. Gender was upstream of both treatment selection and recovery; to get at the real relationship between treatment and recovery we were forced to condition on it. When the treatment has no graphical parents, we don’t need the secondary variable. When it does (and that parent has a ‘backdoor path’ to the effect of interest), we do.

Another Approach

Another way to think about Simpson’s paradox is through the language of pure statistics. First let’s take a small detour back through the basics.

A joint probability can be easily factored into a product of conditionals. Here’s a fun example — you’re getting a new dog, and you’re interested in their personality! Some of the dogs are (Y)oung, some are (I)ntelligent (and some aren’t), and some are quite (E)nergetic.

Below is the box that represents the distribution of dog personality probabilities. We set the area of the square equal to 1. Drop a pin, and the dog it represents will have the personality traits underneath.

Amongst all the possible dogs, you can see some are young, some are smart, some are energetic, and a few are all three. The likelihood of a random dog with some specific combination of traits can be factored easily — below is a simple example to illustrate how when we’re looking for young, intelligent, and energetic dogs (Y=1, I=1, and E=1).

Factoring a joint probability into the product of conditionals.

Why would you want to factor a joint into conditionals like that? In the context of Simpson’s paradox, we’ll answer that by revisiting our first example above.

In that example, we were looking at the joint probability of (T) Treatment, (G) Gender, and (R) Recovery. We want to query the likelihood of R=1 (a successful recovery) given other combinations of gender and treatment. Let’s factor our joint so we can work with it:

All possible ways to rewrite that joint as a product of conditioned variables.

There are six ways to factor a joint of three variables. However, in this case we are trying to take an observation of relationships (in which it’s okay to say things like ‘the probability someone is a man given they recovered successfully’) and instead query causal relationships between them (‘if we force the probability of treatment to equal 1, what happens to a man’s recovery?’). Here we introduce Pearl’s do-operator.

Instead of p(R|T), an observation about the correlation between treatment and recovery, we actually care about p(R|do(T)) — the probability of a recovery given we causally intervene and force treatment (setting T=1 by force of will). The do operator limits how we can factor our joint, because we must keep do(T) on the right side of the conditional line with respect to recovery to answer a causal question about how treatment affects recovery.

So we’re interested in p(R|G, do(T)). That leaves us with two options:

In the second choice we find p(G|do(T)), a nonsensical relationship. We’re now thinking causally, and this term begs the question ‘What is the probability that your gender changes given you had treatment?’ If gender doesn’t change when we do(T), we can’t use a term like this coherently in a causal setting.

The first equation, on the other hand, doesn’t have logical flaws. We notice that p(do(T)|G) is just 1 — the probability we do(T) conditioned on anything rational will be 1, as that’s the definition of the do operator. We can remove this term and simplify our expression.

Another way to say it is that the likelihood of accepting treatment based on gender is not something we want affecting our assessment of the probability of treatment success, so the term has to go. We know this term is bad news because of our causal intuition. Our conditionals must follow our causal instincts, matching our causal graph such that the probability of child nodes are always (and only) conditioned on their parents.

We’re left with p(R | do(T), G) * p(G). It shows us that the likelihood of recovery is related to the fraction of people in that gender, not the overall population. This gives us the same answer as our causal diagram through a more formal process.

Real Life, Man

Simpson’s Paradox can come up in real life — like in the famously contentious gender bias issue at UC Berkeley. Its real beauty, though, lies in illustrating and introducing the rationale behind causal reasoning in statistics.

None of what we did makes sense in the context of standard statistics, but it does make sense. I hope this introduction to Simpson’s paradox gave you a taste for how to start formalizing your causal logic.

For a long time, statistics has lacked the language to discuss causal reasoning and relationships. Thankfully it seems the world of statistics is actively shifting. With it will come a huge boon for science.