COMMENTHISTORY To fight denial,
study Galileo and
Arendt p.309

CHEMISTRY Three more unsung
women — of astatine
discovery p.311

PUBLISHING As well as ORCID
ID and English, list authors
in their own script p.311

ILLUSTRATION BY DAVID PARKINSEVOLUTION Cooperationand conflict from ants
and chimps to us p.308

Retire statistical significanceValentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories
call for an end to hyped claims and the dismissal of possibly crucial effects.
How do statistics so often lead scientists todeny differences that those not educated instatistics can plainly see? For several generations, researchers have been warned that astatistically non-significant result does not‘prove’ the null hypothesis (the hypothesisthat there is no difference between groups orno effect of a treatment on some measured
outcome)1. Nor do statistically significant
results ‘prove’ some other hypothesis. Suchmisconceptions have famously warped theliterature with overstated claims and, lessfamously, led to claims of conflicts between
studies where none exists.
We have some proposals to keep scientists
from falling prey to these misconceptions.

.
d
e
v
r
e
s
e
r
s
t
h
g
i
r
l
l
A
.
d
e
t
i
m
i
L
e
r
u
t
a
N
r
e
g
n
i
r
p
S
9
1
0
2
©

W

hen was the last time you heard
a seminar speaker claim therewas ‘no difference’ between
two groups because the difference was‘statistically non-significant’?
If your experience matches ours, there’s
a good chance that this happened at thelast talk you attended. We hope that at least
someone in the audience was perplexed if, as
frequently happens, a plot or table showedthat there actually was a difference.

2 1 M A RC H 2 0 1 9 | VO L 5 6 7 | NAT U R E | 3 0 5

PERVASIVE PROBLEM

Let’s be clear about what must stop: weshould never conclude there is ‘no difference’ or ‘no association’ just because a P value
is larger than a threshold such as 0.05

COMMENT

BEWARE FALSE CONCLUSIONS
Studies currently dubbed ‘statistically significant’ and ‘statistically non-significant’ need not be
contradictory, and such designations might cause genuine effects to be dismissed.

‘Significant’ study(low P value)‘Non-significant’ study(high P value)

Decreased effectNo effect

The observed effect (orpoint estimate) is thesame in both studies, sothey are not in conflict,even if one is ‘significant’and the other is not.

Increased effect

.
d
e
v
r
e
s
e
r
s
t
h
g
i
r
l
l
A
.
d
e
t
i
m
i
L
e
r
u
t
a
N
r
e
g
n
i
r
p
S
9
1
0
2
©

3 0 6 | NAT U R E | VO L 5 6 7 | 2 1 M A RC H 2 0 1 9

some quality-control standard). And weare also not advocating for an anythinggoes situation, in which weak evidence
suddenly becomes credible. Rather, and inline with many others over the decades, weare calling for a stop to the use of P values
in the conventional, dichotomous way — todecide whether a result refutes or supports ascientific hypothesis5.

QUIT CATEGORIZINGThe trouble is human and cognitive morethan it is statistical: bucketing results into
‘statistically significant’ and ‘statistically
non-significant’ makes people think that theitems assigned in that way are categoricallydifferent6–8. The same problems are likely toarise under any proposed statistical alternative that involves dichotomization, whether
frequentist, Bayesian or otherwise.
Unfortunately, the false belief thatcrossing the threshold of statistical significance is enough to show that a result is‘real’ has led scientists and journal editors to
privilege such results, thereby distorting theliterature. Statistically significant estimatesare biased upwards in magnitude and potentially to a large degree, whereas statistically
non-significant estimates are biased downwards in magnitude. Consequently, any discussion that focuses on estimates chosen for
their significance will be biased. On top ofthis, the rigid focus on statistical significanceencourages researchers to choose data andmethods that yield statistical significance forsome desired (or simply publishable) result,or that yield statistical non-significance foran undesired result, such as potential side
effects of drugs — thereby invalidatingconclusions.
The pre-registration of studies and acommitment to publish all results of all
analyses can do much to mitigate theseissues. However, even results from pre-registered studies can be biased by decisions
invariably left open in the analysis plan9.
This occurs even with the best of intentions.
Again, we are not advocating a ban on
P values, confidence intervals or other statistical measures — only that we should
not treat them categorically. This includes
dichotomization as statistically significant or
not, as well as categorization based on otherstatistical measures such as Bayes factors.
One reason to avoid such ‘dichotomania’is that all statistics, including P values andconfidence intervals, naturally vary fromstudy to study, and often do so to a surprising degree. In fact, random variation
alone can easily lead to large disparities in
P values, far beyond falling just to either sideof the 0.05 threshold. For example, even if
researchers could conduct two perfect
replication studies of some genuine effect,each with 80% power (chance) of achieving
P < 0.05, it would not be very surprising for
one to obtain P < 0.01 and the other P > 0.30.

SOURCE: V. AMRHEIN ET AL.

Association released a statement in The
American Statistician warning against themisuse of statistical significance and P values. The issue also included many commentaries on the subject. This month, a special
issue in the same journal attempts to pushthese reforms further. It presents more than
40 papers on ‘Statistical inference in the 21st
century: a world beyond P < 0.05’. The editors introduce the collection with the caution “don’t say ‘statistically significant’”3.
Another article4 with dozens of signatories
also calls on authors and journal editors todisavow those terms.
We agree, and call for the entire conceptof statistical significance to be abandoned.
We are far fromalone. When we“Eradicating
invited others tocategorization
will help to halt read a draft of thiscomment and sign
overconfident
their names if theyclaims,concurred with ourunwarranted
message, 250 did so
declarations of within the first 24‘no difference’hours. A week later,
and absurd
we had more thanstatements
800 signatories — all
about
checked for an aca‘replication
demic affiliation or
failure’.”other indication ofpresent or past workin a field that depends on statistical modelling (see the list and final count of signatories
in the Supplementary Information). These
include statisticians, clinical and medical
researchers, biologists and psychologists
from more than 50 countries and across allcontinents except Antarctica. One advocate
called it a “surgical strike against thoughtless testing of statistical significance” and “anopportunity to register your voice in favourof better scientific practices”.
We are not calling for a ban on P values.
Nor are we saying they cannot be used asa decision criterion in certain specialized applications (such as determining
whether a manufacturing process meets

or, equivalently, because a confidence
interval includes zero. Neither should weconclude that two studies conflict because
one had a statistically significant result andthe other did not. These errors waste research
efforts and misinform policy decisions.
For example, consider a series of analysesof unintended effects of anti-inflammatory
drugs2. Because their results were statistically
non-significant, one set of researchers concluded that exposure to the drugs was “notassociated” with new-onset atrial fibrillation(the most common disturbance to heart
rhythm) and that the results stood in contrast to those from an earlier study with astatistically significant outcome.
Now, let’s look at the actual data. The
researchers describing their statistically
non-significant results found a risk ratioof 1.2 (that is, a 20% greater risk in exposed
patients relative to unexposed ones). They
also found a 95% confidence intervalthat spanned everything from a triflingrisk decrease of 3% to a considerable riskincrease of 48% (P = 0.091; our calculation). The researchers from the earlier, statistically significant, study found the exact
same risk ratio of 1.2. That study was simply more precise, with an interval spanningfrom 9% to 33% greater risk (P = 0.0003; ourcalculation).
It is ludicrous to conclude that thestatistically non-significant results showed“no association”, when the interval estimate
included serious risk increases; it is equally
absurd to claim these results were in contrast
with the earlier results showing an identicalobserved effect. Yet these common practices
show how reliance on thresholds of statistical significance can mislead us (see ‘Beware
false conclusions’).
These and similar errors are widespread.
Surveys of hundreds of articles have foundthat statistically non-significant results areinterpreted as indicating ‘no difference’ or‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information).
In 2016, the American Statistical

SOURCE: V. AMRHEIN ET AL.

COMMENT
Whether a P value is small or large, caution
is warranted.
We must learn to embrace uncertainty.
One practical way to do so is to rename confidence intervals as ‘compatibility intervals’
and interpret them in a way that avoids overconfidence. Specifically, we recommend that
authors describe the practical implicationsof all values inside the interval, especiallythe observed effect (or point estimate) andthe limits. In doing so, they should remember that all the values between the interval’s
limits are reasonably compatible with thedata, given the statistical assumptions used
to compute the interval7,10. Therefore, singling out one particular value (such as thenull value) in the interval as ‘shown’ makes
no sense.
We’re frankly sick of seeing such nonsensical ‘proofs of the null’ and claims of
non-association in presentations, research
articles, reviews and instructional materials.
An interval that contains the null value willoften also contain non-null values of highpractical importance. That said, if you deem
all of the values inside the interval to be practically unimportant, you might then be able
to say something like ‘our results are mostcompatible with no important effect’.
When talking about compatibility intervals, bear in mind four things. First, justbecause the interval gives the values mostcompatible with the data, given the assumptions, it doesn’t mean values outside it areincompatible; they are just less compatible.
In fact, values just outside the interval do notdiffer substantively from those just inside
the interval. It is thus wrong to claim that aninterval shows all possible values.
Second, not all values inside are equallycompatible with the data, given the assumptions. The point estimate is the most compatible, and values near it are more compatible
than those near the limits. This is why weurge authors to discuss the point estimate,even when they have a large P value or a wide
interval, as well as discussing the limits ofthat interval. For example, the authors above
could have written: ‘Like a previous study,
our results suggest a 20% increase in riskof new-onset atrial fibrillation in patients
given the anti-inflammatory drugs. Nonetheless, a risk difference ranging from a 3%
decrease, a small negative association, to a
48% increase, a substantial positive association, is also reasonably compatible with ourdata, given our assumptions.’ Interpretingthe point estimate, while acknowledging
its uncertainty, will keep you from making
false declarations of ‘no difference’, and frommaking overconfident claims.
Third, like the 0.05 threshold from which
it came, the default 95% used to computeintervals is itself an arbitrary convention. It
is based on the false idea that there is a 95%
chance that the computed interval itself contains the true value, coupled with the vague

WRONG INTERPRETATIONS

An analysis of 791 articles across 5 journals*
found that around half mistakenly assume
non-significance means no effect.
Appropriatelyinterpreted
49%

Wronglyinterpreted
51%

ARTICLES

791*Data taken from: P. Schatz et al. Arch. Clin. Neuropsychol. 20,
1053–1059 (2005); F. Fidler et al. Conserv. Biol. 20, 1539–1544(2006); R. Hoekstra et al. Psychon. Bull. Rev. 13, 1033–1037 (2006);
F. Bernardi et al. Eur. Sociol. Rev. 33, 1–15 (2017).

feeling that this is a basis for a confident
decision. A different level can be justified,depending on the application. And, as in theanti-inflammatory-drugs example, interval
estimates can perpetuate the problems ofstatistical significance when the dichotomization they impose is treated as a scientificstandard.
Last, and most important of all, be
humble: compatibility assessments hingeon the correctness of the statistical assumptions used to compute the interval. In practice, these assumptions are at best subject toconsiderable uncertainty7,8,10. Make theseassumptions as clear as possible and test theones you can, for example by plotting yourdata and by fitting alternative models, andthen reporting all results.
Whatever the statistics show, it is fine tosuggest reasons for your results, but discussa range of potential explanations, not justfavoured ones. Inferences should be scientific, and that goes far beyond the merely
statistical. Factors such as backgroundevidence, study design, data quality andunderstanding of underlying mechanisms
are often more important than statisticalmeasures such as P values or intervals.
The objection we hear most against
retiring statistical significance is that it isneeded to make yes-or-no decisions. But
for the choices often required in regulatory, policy and business environments,
decisions based on the costs, benefits andlikelihoods of all potential consequences
always beat those made based solely on
statistical significance. Moreover, for decisions about whether to pursue a researchidea further, there is no simple connection
between a P value and the probable resultsof subsequent studies.
What will retiring statistical significancelook like? We hope that methods sections

and data tabulation will be more detailed
and nuanced. Authors will emphasize theirestimates and the uncertainty in them — forexample, by explicitly discussing the lower
and upper limits of their intervals. They will
not rely on significance tests. When P values
are reported, they will be given with sensible
precision (for example, P = 0.021 or P = 0.13)— without adornments such as stars or letters to denote statistical significance and not
as binary inequalities (P < 0.05 or P > 0.05).
Decisions to interpret or to publish results
will not be based on statistical thresholds.
People will spend less time with statisticalsoftware, and more time thinking.
Our call to retire statistical significanceand to use confidence intervals as compatibility intervals is not a panacea. Although it
will eliminate many bad practices, it could
well introduce new ones. Thus, monitoring
the literature for statistical abuses should be
an ongoing priority for the scientific community. But eradicating categorization will
help to halt overconfident claims, unwarranted declarations of ‘no difference’ andabsurd statements about ‘replication failure’
when the results from the original and replication studies are highly compatible. The
misuse of statistical significance has donemuch harm to the scientific community andthose who rely on scientific advice. P values,
intervals and other statistical measures all
have their place, but it’s time for statisticalsignificance to go. ■Valentin Amrhein is a professor of zoology
at the University of Basel, Switzerland.
Sander Greenland is a professor of
epidemiology and statistics at the
University of California, Los Angeles. Blake
McShane is a statistical methodologist andprofessor of marketing at Northwestern
University in Evanston, Illinois. For a fulllist of co-signatories, see Supplementary
Information.
e-mail: v.amrhein@unibas.ch
1. Fisher, R. A. Nature 136, 474 (1935).
2. Schmidt, M. & Rothman, K. J. Int. J. Cardiol. 177,
1089–1090 (2014).
3. Wasserstein, R. L., Schirm, A. & Lazar, N. A.
Am. Stat. https://doi.org/10.1080/00031305.20
19.1583913 (2019).
4. Hurlbert, S. H., Levine, R. A. & Utts, J. Am. Stat.
https://doi.org/10.1080/00031305.2018.1543
616 (2019).
5. Lehmann, E. L. Testing Statistical Hypotheses 2nd
edn 70–71 (Springer, 1986).
6. Gigerenzer, G. Adv. Meth. Pract. Psychol. Sci. 1,
198–218 (2018).
7. Greenland, S. Am. J. Epidemiol. 186, 639–645(2017).
8. McShane, B. B., Gal, D., Gelman, A., Robert, C. &Tackett, J. L. Am. Stat. https://doi.org/10.1080/0
0031305.2018.1527253 (2019).
9. Gelman, A. & Loken, E. Am. Sci. 102, 460–465(2014).
10. Amrhein, V., Trafimow, D. & Greenland, S. Am.
Stat. https://doi.org/10.1080/00031305.2018.1
543137 (2019).
Supplementary information accompanies thisarticle; see go.nature.com/2tc5nkm

.
d
e
v
r
e
s
e
r
s
t
h
g
i
r
l
l
A
.
d
e
t
i
m
i
L
e
r
u
t
a
N
r
e
g
n
i
r
p
S
9
1
0
2
©

2 1 M A RC H 2 0 1 9 | VO L 5 6 7 | NAT U R E | 3 0 7