P-values are less diagnostic than we like to think
By: Tadeg Quillien
Tags: Probability, P Value, Bayes Theorem, Statistics

A significant p-value is less ground for confidence than we would like to think. The problem is that we spontaneously interpret p-values through a “Bayesian” lens, where they do not really fit.

First, many people think that a p-value tells you the probability that your hypothesis is false. But this is wrong, notably because p-values suffer from base rate neglect (they can lead you to accept even hypotheses that are abysmally implausible a priori).

This post is about a second, more subtle problem.

For a long time I thought that, in cases where base rates are not a problem, then surely p-values have a straightforward interpretation. Surely, if before collecting the data you think that H0 and H1 are equally plausible, and the data give you p < .05, then you should think that there is a more-than-95% chance that H1 is right.

But this is not correct. The reason is that the quantity expressed by the p-value is strangely irrelevant to statistical inference.

—

Your friend Mark has two coins in a box. One of them is perfectly fair, but the other coin has a bias: it lands Heads 61% of the time. Mark shuffles the box, and draws one coin randomly from the box.

Then he tosses the coin 100 times. It lands Heads 61 times. How confident can you be that the coin is the biased one?

Let’s see if p-values suggest an answer.

Here, a p-value gives the probability that we get a number of Heads as extreme, or more extreme, than 61, assuming that H0, the null hypothesis, is true (H0 is “the coin is fair”). To compute the p-value, I take a fair coin, toss it 100 times, and count the number of Heads. I repeat this a million times (using R), counting the number of Heads I get each time.

I get the following distribution:

Then I simply count the number of simulations where I get a value that is as extreme (or more extreme) than 61. That is, all simulations where the number of Heads is either lower than 40, or higher than 60. This is the blue area in this histogram:

I get such an extreme result in 3.5% of the simulations. So, p = .035.

Where do we go from there?

It feels natural to view the p-value as a measure of how well the data fits with the null hypothesis. To determine how much the data favors H1 over H0, we could compare the p-value with an analogous measure that tells us how well the data fits H1.

How well does the data fit with the hypothesis that the coin is biased? Given that the biased coin has a 61% bias toward Heads, then every possible outcome is “as extreme or more extreme” than actually observing 61 Heads in 100 tosses:

Therefore it feels natural to say that the likelihood of the data under H1 is 1.

Now we can try to quantify how much the data favors H1 over H0. We simply take the ratio of the likelihood of the data under H1 and under H0, and we obtain 1 / .035 = 28.6. I.e., we conclude that the data we obtained are 28.6 times more likely if the coin is biased than if the coin is fair. Both hypotheses were equally likely a priori, so we do not need to worry about base rates. Therefore we conclude that we are now 28.6 times more confident in H1 than H0.

But this is wrong. As we’re about to see, this calculation over-estimates the actual strength of the evidence by a factor of almost 2.5!

The correct computation

Bayes’ rule tells us how to get the correct answer. First we need to count how many times we got exactly 61 Heads, in the simulations where we used a fair coin. This corresponds to the red bar in this histogram:

Then we need to count how many times we got exactly 61 Heads, in the simulations where we used a biased coin:

The next thing Bayes’ rule tells us is very intuitive: the strength of evidence in favor of H1 is simply the number of times we got exactly 61 Heads with the biased coin, divided by the number of times we got exactly 61 Heads with the fair coin. I.e., it is the size of the large red bar, divided by the size of the small red bar.

Out of 1 million simulations with the biased coin, I got 61 Heads 82012 times. Out of 1 million simulations with the fair coin I got 61 Heads 7064 times. This means that the strength of the evidence is 82013/7064 = 11.6.

Quite less than the 28.6 we found before! What went wrong the first time?

The problem

The problem is that the p-value does not give us what we want.

Bayes’ rule tells us that what matters is P(d|h): the probability that, assuming hypothesis h is true, we would get the exact same data we actually observed. But the p-value does not give us P(d|h). Instead it gives us the probability of getting data that is as extreme (or more extreme) than the data we actually observed. And Bayes’ rule tells us that this quantity is irrelevant to statistical inference. There is no good normative reason we should care about how “extreme” the data is.

In pictures: the really interesting information is this:

But the p-value looks at this:

The p-value’s concern with extreme values unfairly disfavors H0. The quantity that is relevant to evaluating H0 is the probability of getting exactly 61 Heads. But in evaluating H0, the p-value also looks at the probability of getting 62, or 63, or 64 (and so forth up to 100) Heads. These irrelevant probabilities make H0 look bad, because they are very small.

Note that in our example, H1 (“the coin is 61% biased toward Heads”) is the hypothesis that is maximally compatible with the data. With any other H1, a p-value of .035 would be even less diagnostic. For example, if H1 is “the coin’s bias toward Heads is somewhere between 51 and 100%”, then observing 61 Heads in 100 tosses gives us p = .035, but makes us only 2.7 times more confident in H1 than H0.

Conclusion

Presumably, scientists collect data because they are interested in P(h|d), the probability that a hypothesis is true, given the data. So one might expect that p-values tell us something especially relevant to P(h|d).

But they don’t really do that. The formula that tells us how to compute P(h|d) is Bayes’ rule. You can squint as hard as you want at Bayes’ rule, you will not find anything like the p-value among its building blocks.

This is a problem because most people may spontaneously interpret p-values in Bayesian terms. Even people who have never heard about Bayes! It’s just that Bayesian interpretations of statistical values are intuitively very compelling. For instance many people think that the p-value gives 1-P(h1|d). Even if you know how to avoid that mistake, you may spontaneously and implicitly think that the p-value quantifies P(d|h0). But as we saw above, it doesn’t, and thinking it does may lead you to vastly overestimate how much the data favors one hypothesis over another.

Of course, this doesn’t mean that p-values tell us nothing about our hypotheses. My point is that what they do tell us is not straightforward.

The problems with p-values don’t mean we should necessarily stop using them — there might be good reasons why they work the way they do. But it is good to be aware of these problems, lest we put too much confidence in our statistical results.

References:

This post is inspired by pp. 63–64 in Chapter 3 in MacKay’s book Information Theory, Inference, and Learning Algorithms (2003, Cambridge University Press), available at https://www.inference.org.uk/itprnn/book.pdf.

Paul Smaldino’s Gene Clicker tutorial is a great introduction to the base rate neglect problem with p-values: http://xcelab.net/boffins/

Evidence for systematic misinterpretations of p-values can be found in (e.g.) Oakes (1986); Cassidy et al. (2019).

Appendix: explanation of the correct computation

Bayes’ rule says that the probability that a hypothesis h is correct, given the data we observed, is given by:

P(h|d) = P(d|h)P(h) / P(d)

So we have

P(h0|d) = P(d|h0)P(h0) / [P(d|h0)P(h0) + P(d|h1)P(h1)]

and

P(h1|d) = P(d|h1)P(h1) / [P(d|h0)P(h0) + P(d|h1)P(h1)]

To compute “How probable is H1, relative to H0”?, we compute P(h1|d)/P(h0|d)

Since P(h0) and P(h1) are equal to each other (before seeing the data we think that there is a 50% chance that the coin is biased), we have:

P(H1|d) / P(H0|d) = P(d|H1) / P(d|H0)