Statistics Commentary Series: Commentary No. 27: P-Hacking.

R esearchers live and die by the famous (or infamous) cut-off of P ≤ 0.05 – if the results of the study show a P level less than or equal to 0.05, then we can submit the paper and fame, promotion, and tenure will be ours; larger than 0.05, and will we have nothing to show for our time and effort except heartbreak, despondency, and a file drawer full of useless data. Although many have often pointed out the arbitrary nature of this criterion and have called for its demise, it, like the phoenix, keeps rising from the ashes, only to live again. (Others have likened it tomosquitos – annoying and impossible to swat away.) There aremany legitimate thingswe can do tomaximize our chances of getting significant results: design and execute the study well, use reliable and valid outcome measures, calculate the sample size beforehand to insure adequate power, use the appropriate statistics, and so forth. And, with so much resting on crossing that magical threshold, it is almost fatuous to say that people have come up with not so legitimate methods, of which “P-hacking” (a term introduced by Simmons, Nelson, and Simonsohn) is perhaps the most wide-spread (we’ll leave aside faking the data and plagiarism, as they are totally beyond the pale, but do in fact represent the reasons the majority of articles are retracted). P-hacking – the relentless search for statistical significance – comes in many guises. The most pervasive form is also referred to as “data dredging,” “significance chasing,” “data snooping,” or “going on a fishing expedition,” and refers to combing throughmasses of data, looking for significance, most often without any hypotheses guiding the search. Although data dredging has been with us ever since people measured more than twovariables at a time, it got a boost (if that’s the correct term) with the convergence of desk-top computers, “big data” amassed by corporations, and the ability to collect thousands of variables simultaneously, as with the human genome or computerized scans of the brain. The reason it is highly suspicious goes back to a point discussed in a previous article. If the null hypothesis is true (ie, nothing is going on), and we run k tests with an alpha level of 0.05, then the probability of finding at least one significant finding (P1) is given by the formula: P1 = 1 – 0.95 . Run 10 tests and P1 is 40%; with 20 it is 64%; and with 100 tests, it is greater than 99%; that is, if you do enough tests, you are virtually guaranteed that something will emerge as significant. This was delightfully shown in an article by Puget, who conclusively showed that green dice are loaded. He (or, more accurately, his computer) rolled 10 green dice 100 times. It would be expected that the number 6 would appear 1,000/ 6 = 167 times, whereas it actually showed up 188 times. The difference between the expected and observed values is statistically significant (P = 0.04). Had this been the only result he reported, we could be excused if we insist that the casino give us only green dice to roll at the roulette table the next time we visit. However, it was just 1 part of a larger study, which involved doing the same procedure with 20 sets of dice, each of a different color. As it turned out, green was the only color for which there were more 6s than expected. What makes this P-hacking rather than a legitimate finding is that he did not begin his experiment by asking if the results for green dice would be different from those of other colors. Rather, he just amassed a lot of data and, without any hypotheses, searched for statistical significance; not surprisingly, with 20 statistical tests, he found it. Finding spurious effects has been shown in a number of contexts (some of them humorous), ranging from the association of having been born under a specific astrological sign with various medical conditions or responses to therapy, to fMRI patterns to visual stimuli recorded from a dead salmon. There is even a Web site devoted to graphs showing high correlations between totally unrelated variables, such as the per capita consumption of cheese and the number of peoplewho died by becoming tangled in their bedsheets (r = 0.947), the divorce rate inMaine and the per capita consumption of margarine (r = 0.993), and the total revenue generated by arcades and the number of computer science doctorates awarded in the U.S. (r = 0.985), proving not only that if you look hard enough, you will always find something correlated with something else, but also why you can never infer causality from a correlation. A related nefarious way to P-hack is to change the outcome from one that is looking unpromising to another one that looksmore likely toyield significance. Sad to say, this appears to be a pervasive problem