Insignificance of statistical significance? Why the p-value should not be overrated.
By: Hannah Wnendt
Tags: Data Science, P Value, Statistics, Hypothesis Testing, Analytics

An imaginary case study — study extraterrestrial life in space

Imagine you are part of a research team who has discovered nonhuman intelligence on plant X that does not seem to be dangerous to humans. Eager to find out more about this species, you instruct your colleagues to collect data about their some of their main characteristics. Apart from cellular samples and behavioural observations, you send a bunch of space-proof robots to collect data from their physical height as they have approximately the same height as humans. The glibberish texture of their skin makes you name them glibglobs in the lab — just like some creatures are called in your favourite cartoon series named Rick and Morty. Now your colleagues have noticed that glibglobs have different sexes — one of them being a bit taller than the other. As females are usually smaller in height than males on earth, you are starting to wonder: Is this sex-dependent difference maybe a universal pattern in our universe? Okay — such a big question, so let us make it more specific: Are glibs smaller than their globs? Or has this been just a random observation by one of your colleagues?

As you have collected 100 height samples for each of the sexes, you start to study this phenomenon visually and ask yourself if there really is a fundamental difference between glibs and globs. Looking at the plots, you can see globs are on average taller than glibs, but there is also a lot of variation going on. In some case, glibs are even taller than globs — so maybe there is even no difference between the sexes if you would have the resources to measure every glibglob on whole planet X. How can you tell how likely your data is assuming there is maybe even no difference in height at all?

Density plot for glibglob height data (simulated in R), each sex is coloured (red = glibs, blue = globs)

After you have checked whether the data is appropriate for the test that you would like to use to answer this question, you run a computer program to compare the means of both sexes. There are two possible scenarios:

1. The true difference in height is equal to 0 in the population.

2. The true difference in height is NOT equal to 0 in the population.

You can imagine the first hypothesis (called „null-hypothesis“) as a kind of default in significance testing — that is probably the most confusing thing at first sight because you are actually interested in the second one. Now imagine a part of your computer program is a critical sparring partner which suggests that there is no difference. Rational as it is, it suggests that chance is operating alone. Still you assume that there might be something different going on, backed up with some real glibglob data and present your alternative to this view. Let’s debate! To get a bit closer to the answer, the computer runs some calculations for you. It now “pretends” to run the same experiment over and over again and thereby generates lots and lots of simulated samples of imaginary glib glob data. These fake samples represent how the height difference could look like if glibs and globs actually had the same height, allowing for some random variation. Now the computer generates a summary statistic (e.g., a t-statistic) for a) your observed sample data as well as for b) each simulated sample (the fake data).

Experience shows that as long as we have many observations, a lot of phenomena in nature approximate a bell-shaped distribution, called normal distribution. This also applies to the fake data of theoretically possible glibglob height differences: If you take a random sample out of the fake data, you will more likely observe height differences around zero, allowing for a little random variation. On the other hand, it will be less likely to observe extreme differences that just occur by chance. Still not sure about your hypothesis that there could be a difference in height, your computer finally compares this observed summary statistic against the summary statistic of the fake samples it has generated earlier and tells you where your data would fall onto the distribution of fake samples that are completely random. Giving you a p-value < 0.01, the computer must admit that your sample makes his default state („there is no difference in glibglob height!“) look a bit unusual. Why? Because there seems to be unlikely to observe this or even more extreme data when assuming that chance was operating alone.

The p-value estimates the probability to observe such or more extreme data given that there is actually no effect in nature — assuming that your model suits the data and you stick to your original intentions.

So, what can that p-value tell me? And what not?

Congratulations for coming this far. Look at that lengthy definition. Can that p-value now answer your original question?

Can you tell your colleagues that it is likely that there is a true difference in height between glibglobs by nature, given that you have observed these data?

Nope. Confusing, right? Although this is actually the question we are interested in, this question can only answered using Bayesian Statistics which encompasses a completely different way of thinking and mathematical approach (Dienes & Mclatchie, 2018; Kruschke & Liddell, 2018a, 2018b; Vandekerckhove, Rouder, & Kruschke, 2018). If people using traditional significance testing claim that their hypothesis is true because null hypothesis was rejected, this should raise some red flags to you. Taken by itself, a small p-value does indicate that for now, the data seems to be incompatible with a scenario in which there is actually no effect to begin with. But it cannot tell you if your proposed alternative (e.g., „there is a sex-dependent difference in glibglob height greater than zero.“) is likely to be true because it does not incorporate any prior knowledge (e.g., how likely is this alternative in the first place?) — this would have been the posterior probability of the alternative which got updated by our fresh glibglob data, but this is beyond the scope of the current article.

2. Can you tell your colleagues that the data seems a bit interesting considering the claim that there is NO true difference in height in glibglobs?

Yes, you can. If you have checked the preconditions required by the model and strictly kept your sampling intentions constant during the experiment (e.g., stopped measuring after collecting a predefined sample size regardless of the outcome of your experiment), this height phenomenon is maybe worth investigating further. Given that nobody has measured these creatures before and your experiment is thus quite exploratory, this result provides a preliminary starting point for ongoing research on the topic.

3. Can you tell your colleagues that the results are practically meaningful?

Nope. You have no idea if the difference in height between glibglobs serves any role in their life on planet X and cannot even tell solely based on that p-value whether that difference is equally large in magnitude like the one observed in humans. Likewise, you cannot tell if you have asked the right question. Maybe there are some characteristics of glibglobs that are way more fascinating studying which you have not considered so far?

Now you may feel a bit disappointed. You have come such a long way to planet X, measured a bunch of glibglobs and still know so little about them — it seems to be just the start of an even longer journey of various research teams taking a closer look at that foreign species. Knowing that you are part of the first research team studying glibglobs, you are still tapping in the dark, even if you have a tiny little signal pointing towards a specific direction.

Okay — this story probably seems to be a bit far-fetched and oversimplified. Nevertheless, it hopefully illustrates the traditional approach towards statistical significance in a concise manner and understandable way. If you are still struggling, don’t worry: Even scientists or health professionals usually have trouble interpreting the p-value (Gigerenzer, 2004; Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz, & Woloshin, 2007; Gigerenzer & Marewski, 2015). So, it should not be surprising if you are still stuck in your head.

So what?

Basically, we can say goodbye to the idea to answer questions about nature very quickly, effortlessly and precisely. Therefore, it may take a lot of perseverance and tolerance to a) wrap your head around this, b) be able to endure your curiosity and c) forget the dream of having one magical universal statistical tool applicable to all problems. To make meaningful conclusions about complex phenomena, we need to purposefully picking from a whole statistical toolbox each time we solve a new problem. It takes a lot of training for the analyst not only to be able to run the analysis, but also ask the right question in the first place, choose a suitable way to test it and finally interpret the findings. When it comes to human behaviour, it should become even more apparent that it takes a lot to measure it reliably and make general inferences about the sum of manifold individuals. This is why experts suggest to always explicitly communicate the uncertainty that accompanies the research findings, provide a detailed documentation about every single step that led to the result and share the data with the research community. This makes scientific reasoning more traceable, demonstrates whether or not an effect holds true across different Laboratories and paves the way for future researchers to answer key questions altogether. By integrating evidence in respect to phenomena of interest, we are able to paint a bigger picture that probably represents its nature more precisely (Hunter & Schmidt, 2004; Weinberg, 2001).

The p-value has been criticised to divide study results in significant and non-significant, thus sadly and erroneously considered to be not worth a publication for many research journals. Because there are clear cut-off values for the p-value for a result to be considered statistically significant (usually p < .05 or p < .01), it supports a black-or-white style of thinking. Moreover, significance testing is only the end result of a long chain of other decisions the researcher makes beforehand (e.g. the research design, sampling method, model selection etc.) which determine the quality of the research but this simply cannot get expressed into a single value. However, the p-value taken by itself is a useful tool to indicate non-random patterns in the data and — if used thoughtfully — allows statistical inference to become fast and scalable.

I would never claim that science is nonsense because of this practice — the p-value just needs to be used and interpreted as one of many tools and not put as a mindless decision-making cut-off. This is exactly what the inventor of p-value, Sir Ronald Fisher, aimed for — it should be taken as a hint that indicates whether your observations are worth a closer look. Nevertheless, the p-value is not a fool-proof concept, seems to be quite a bit counterintuitive and therefore does require in-depth training of the analyst to interpret the results with the right amount of scepticism. Fisher (1956) even writes: „No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.’’ Consequently, it is not the p-values fault because it just does what it should. We just should not expect too much from it or yield to the temptation to switch off our minds.

Author: Hannah Wnendt, 01.09.202

References

Dienes, Z., & Mclatchie, N. (2018). Four reasons to prefer Bayesian analyses over significance testing. Psychonomic Bulletin and Review, 25(1), 207–218. https://doi.org/10.3758/s13423-017-1266-z

Fisher RA. Statistical methods and scientific inference. Edinburgh: Oliver and Boyd; 1956.

Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33(5), 587–606. https://doi.org/10.1016/j.socec.2004.09.033

Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, Supplement, 8(2), 53–96. https://doi.org/10.1111/j.1539-6053.2008.00033.x

Gigerenzer, G., & Marewski, J. N. (2015). Surrogate Science: The Idol of a Universal Method for Scientific Inference. Journal of Management, 41(2), 421–440. https://doi.org/10.1177/0149206314547522

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3

Ioannidis, J. P. A. (2018). Why most published research findings are false. Getting to Good: Research Integrity in the Biomedical Sciences, 2(8), 2–8. https://doi.org/10.1371/journal.pmed.0020124

Kruschke, J. K., & Liddell, T. M. (2018a). Bayesian data analysis for newcomers. Psychonomic Bulletin and Review, 25(1), 155–177. https://doi.org/10.3758/s13423-017-1272-1

Kruschke, J. K., & Liddell, T. M. (2018b). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin and Review, 25(1), 178–206. https://doi.org/10.3758/s13423-016-1221-4

Peng, R. (2015). The reproducibility crisis in science: A statistical counterattack. Significance, 12(3), 30–32. https://doi.org/10.1111/j.1740-9713.2015.00827.x

Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Sage.

Vandekerckhove, J., Rouder, J. N., & Kruschke, J. K. (2018). Editorial: Bayesian methods for advancing psychological science. Psychonomic Bulletin and Review, 25(1), 1–4. https://doi.org/10.3758/s13423-018-1443-8

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s Statement on p-Values: Context, Process, and Purpose. American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

Weinberg, C. R. (2001). It’s time to rehabilitate the P-value. Epidemiology, 12(3), 288–290.