# An open letter to evolutionary and human sciences; Statistics has moved on and so should we. A proposal for more transparent research, and some notes on p < 0.003

Lloyd A Courtenay

► To cite this version:

Lloyd A Courtenay. An open letter to evolutionary and human sciences; Statistics has moved on and so should we. A proposal for more transparent research, and some notes on p < 0.003. Quaternary Environments and Humans, 2024, 2 (6), pp.100041. 10.1016/j.qeh.2024.100041. hal-05329930

HAL Id: hal-05329930

https://hal.science/hal-05329930v1

Submitted on 24 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An open letter to evolutionary and human sciences; Statistics has moved on and so should we. A proposal for more transparent research, and some notes on *p* < 0.003

Lloyd A. Courtenay<fnref n="1" />

CNRS, PACEA UMR 5199, Université de Bordeaux, Bât. B2, Allée Geoffroy Saint Hilaire, CS50023, Pessac 33600, France

<figure><img src="image_2.png" /></figure>

## ARTICLE INFO

Keywords:

p-Values

Statistical Reasoning

Statistical Inference

Epistemology

## ABSTRACT

Statistical reasoning and inference have become integral to how we reach conclusions as scientists. Few papers in evolutionary, biological, and human sciences, are published at present without at least one report of a p-value, or the term "statistically significant". As a product of this, discoveries and results often hinge on p-values below the infamous threshold of 0.05. This is due to how well ingrained these notions are in many of our higher education systems, and our practices as researchers. Nevertheless, the concept of a p-value, borrowed from statistics, and dating as far back as the late 20th and early 21st century, has undergone an evolution that we may not be completely aware of, if we do not follow or keep ourselves up to date with the most recent advances in statistical research. The present short communication can be framed as an opinion piece that simply aims to call our attention to the fact that statisticians, in recent years, have asked that we completely abandon the notion of "statistically significant (p < 0.05)". It is important to point out that none of these observations are new, however certainly merit being re-addressed. Here I attempt to open a dialogue among archaeologists, palaeontologists, palaeoanthropologists, biological anthropologists, evolutionary biologists, and all researchers in related fields, about such statistical research. Finally, I propose a means of moving forward, suggesting we move from a world where p < 0.05 is considered a binary threshold for conclusive results, to a more nuanced practice where different gradients, down to p < 0.003, are interpreted as increasingly suggestive evidence – provided that findings are supported by additional, transparently reported corroborative data.

## 1. Introduction

In the majority of the social and earth sciences, researchers frequently adopt methods, theories, and concepts from other fields, to facilitate and advance our own lines of investigation. This practice enhances the inferences we can make, especially about the past, as well as the evolutionary mechanisms that have brought us to our current state. This interdisciplinary, and sometimes transdisciplinary, approach, is not unique to this domain; many fields of science incorporate and evolve methods from other disciplines. Archaeology, in particular, exemplifies a field that benefits significantly from external methodologies; however, it often lags in adapting to subsequent advancements as these

originating disciplines evolve.

All fields of science evolve, and this evolution is far from linear (Kuhn, 1962; Feyerabend, 1993). Epistemologically, theoretical and practical evolution is a necessity, and is conditioned by our discoveries and needs along the way. However, one of the most striking examples of a discipline's inability to keep pace with parallel evolving lines of enquiry is the application of statistical reasoning to the archaeological, palaeontological, palaeoanthropological, and even biological, record.

Over the years, numerous instances can be found in which parametric tests, such as Analysis of Variance (ANOVA), the Pearson correlation coefficient, the *t*-test, and the $\chi^2$ test, are employed without meeting the underlying assumptions defined by statisticians; typically,

<footnote>E-mail address: ladc1995@gmail.com.</footnote>

<footnote n="1">¹ ORCID ID: 0000-0002-4810-2001</footnote>
that of normality, homogeneity, linearity, or that the data fits to a Gaussian distribution. This can even be extended to the calculation of basic descriptive measures, such as the mean (μ), standard deviation (σ), or the Coefficient of Variation (CV), that are also unreliable when exposed to non-Gaussian distributions. Similarly, there are cases where calculations are erroneously applied to unsuitable variables, such as the use of the aforementioned approaches and tests on circular variables, such as directions or angles.<fnref n="2" />

While many researchers in our (and all related) fields have a back-
ground heavily rooted in humanities, and for many of us the last expo-
sure we may have had to mathematics was long before university-level
education, it is fundamental that we understand that the techniques we
are using are fundamentally rooted in mathematical reasoning and
equations. It is also important to note that, while the development of
software and computational tools that are open access and easy to use
present considerable advancements for the applicability and replica-
bility of these methods (Marwick, 2016; Marwick et al., 2018), these
tools can also present a number of drawbacks if not used carefully (see
notes and observations by Courtenay, 2023a). Many software function
as black boxes, allowing us to apply them to data without fully under-
standing their appropriateness. Some similar examples can be found
when blindly applying R code and functions without looking at what is
going on underneath. Computers will execute the commands we give
them without questioning the validity of our instructions. It is the job of
the researcher, however, to ensure we understand, at least in theory, the
functioning of these calculations.

The objective of the present “open letter” is to open a dialogue about
one of the most important updates that our field requires; the means in
which we accept or reject hypotheses based on statistical reasoning.
Since the beginning of the use of statistics in our discipline, many re-
searchers have been using *p* < 0.05 as a means of accepting a theory,
without considering what this abstract series of characters, symbols, and
numerals, truly represents. Recent advances and debates, however,
show many statisticians to have stopped using this rule a while ago. In
some instances, practitioners and authors in ecological and evolutionary
sciences have also pointed out this issue (e.g., Smith, 2017, 2019),
however, a lot of the community is still unaware about the existence of
such a debate. The objective here is therefore to redraw our attention to
this issue, and propose a means of moving forward. This should ensure a
more robust and rigorous understanding of fields related to biological
evolution, and in particular, our human evolution.

## 2. What is *p* < 0.05 and where does it come from?

The *p*-value has existed for a while (Kennedy-Schaffer, 2019), with
early appearances of *p*-value-like concepts and varying forms of notation
appearing in the works of some of the most relevant names in statistics;
Laplace (1823), Poisson (1837), Pearson (1895), inter alia. One of the
early appearances of a *p*-value, with a relative definition of a “threshold”
for accepting a hypothesis, appears as far back as the work of Edgeworth
(1885): pg. 185); “by far the greater proportion (namely 0.995)...
Accordingly, if out of a set of [...] N statistical numbers which fulfil the law of
error, we take one at random, it is exceedingly improbable that it will differ
from the Mean to the extent of twice, and à fortiori thrice, the modulus...
such is the nature of the law of error”. However, it is not until the later
works of Pearson (1895), Elderton (1902) and Student (1908), we find

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** – A visual representation of the different statistical means of testing a hypothesis. In this figure we present two distributions, the Null Hypothesis H0 probability distribution, and the Alternative Hypothesis probability distribution. In the model by Fisher (1925), only H0 is considered. In the model by Pearson (1933a, b), both curves are considered. Observations above the solid horizontal line would be considered evidence against H0, and thus reason to accept Hₐ. The area under the H0 curve above this point represents the probability of a Type I statistical error, False Positive, or instances where we reject H0 when it is in fact correct. The area under the Hₐ curve below this point represents the probability of a Type II statistical error, a False Negative – instances where we accept the Null Hypothesis when it is in fact wrong. The dotted vertical line presents the threshold defended here.</figcaption></figure>

valuable insights into early statistical reasoning, building a framework
for what would eventually become a formal definition of the *p*-value by
Fisher (1925).

In Fisher's referential book titled Statistical Methods for Research Workers (Fisher, 1925), Fisher states;

> “The value for which p = 0.05... is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not. Deviations exceeding twice the standard deviation are thus formally regarded as significant” (Fisher, 1925; pg. 47)

Here it is important to point out specifically the phrase “twice the
standard deviation”, from which it can be seen that 0.05 was not an
arbitrarily chosen value, but more theoretically derived from 1 – 2σ ≈
0.05. To many this would be the first instance where this number is
defined, however was not published without some debate. In a series of
studies by Neyman and Pearson (1933a,b), slightly different definitions
are proposed, including;

> "...the testing of statistical hypotheses cannot be treated as a problem of estimation" (pg. 492) "H₀ is accepted when some alternative [Hₐ] is true, the consequences that follow will depend upon the nature of [Hₐ] and its difference from H₀... while it is the 'size' of P₁(w) that matters, it is what may be termed the quality of errors contributing to the risk P₂(w), that must be taken into account" (Neyman and Pearson, 1933a: pg. 497)

> “We deal first with the class of alternatives for which α < α₀. If ε = 0.05... we shall probably decide to accept the hypothesis H₀ as far as this class of alternatives is concerned” (Neyman and Pearson, 1933b: pg. 303)

Once again, we see the value 0.05 appear, however the epistemo-
logical reasoning behind defining this number, and what it represents, is
different. In either case, the concept of a *p*-value considers a reference
distribution that represents what is referred to as the *Null Hypothesis*, or
H₀. In both Fisher (1925)'s and Neyman and Pearson (1933a,b) defini-
tions, the authors discuss a means of accepting or rejecting H₀ by
considering the observation that a given *p*-value, calculated from our
actual data, falls far enough away from the assumptions given by a
theoretical distribution, H₀ (Fig. 1). In Fisher's definition, however, the
author discusses whether the observation is below 1–2σ, with reference
exclusively to the assumptions of H₀, while Neyman and Pearson

<footnote n="2">² Throughout this paragraph I intentionally refrain from citing specific ex-
amples of such practices to avoid targeting individual researchers. The goal
here is not to single out specific instances of malpractice but to encourage all
practitioners in our discipline to remain open-minded. I, too, have made
incorrect calculations in my earlier papers when, as a student, I was unaware of
the limitations of many statistical methods. This is not an excuse, however, to
not acknowledge our shortcomings and continue to perform these tests and
calculations in the future.</footnote>
(1933a,b) are the first to discuss and consider the possibility that an alternative hypothesis (Hₐ) exists. From the latter perspective, the authors also refer to the possibility that accepting H₀ will result in a statistical error, referred to in their text as *P*<sub>I</sub>(*w*) and *P*<sub>II</sub>(*w*), whereby; given we reject H₀ when it is in fact true, we will be committing a Type I statistical error (*P*<sub>I</sub>), while accepting H₀ when it is in fact false will result in a Type II statistical error (*P*<sub>II</sub>) (Fig. 1).

While the two authors propose the exact same number (p = 0.05), the reasoning behind reaching this value is distinct, thus implying that the nature of these thresholds are different as well. This has propagated to years of mechanical use of this number with little consideration to why exactly it was defined. Nevertheless, while many researchers working in the application of statistical inference have taken this value for granted, many studies throughout the years by statisticians themselves have slowly debated the precise reliability of this value, and the means in which it is defined. The precise nature of these studies, however, goes far beyond the scope of this short communication.

As of the 21st century, the definition of a p-value can generally be summarised by the following statement; "a p-value is the probability under a specified statistical model that a statistical summary of the data... would be equal to or more extreme than its observed value" (Wasserstein and Lazar, 2016). In a sense, therefore, p-values indicate how incompatible our empirical data fits with a specified statistical model, however, by itself it does not inform us about the degree to which this hypothesis is true, and can often be misleading without considering the nature of the alternative hypothesis as well. The infamous value of 0.05, on the other hand, was not arbitrarily chosen, and is highly relevant for future discussions on its suitability for applied statistical research.

### 3. What do statisticians say about the topic?

In October, 2017, the American Statistical Association (ASA) held a 2-day symposium to discuss the use of *p*-Values and Statistical Significance, resulting in a large collection of detailed articles (43 in total) that were included in the 73rd volume (Issue Sup1) of the journal *The American Statistician* (Wasserstein et al., 2019). While this collection builds on the original statements proposed by the same association in the same journal, and published in 2016 (Wasserstein and Lazar, 2016), the 2019 Special Issue (SI) is much more extensive and presents a number of fundamental proposals on how to proceed. Nevertheless, it is important to point out that debate on this topic have existed long before this (e.g., Cohen, 1994; Goodman, 2008), yet have had little effect on the presentation of statistical results in other fields.

Among the many papers included within this SI, a common thread and point of concern raised by multiple authors is the predominantly Frequentist<fnref n="3" /> perspective many scientists have when approaching hypothesis testing. Additionally, authors emphasise the need to transition away from the dichotomous black-and-white view of statistical results as simply “significant” or “non-significant”, advocating instead for a more nuanced, gradational approach to interpreting results. From this perspective, multiple authors proposed that a means of overcoming the issue with *p*-value interpretation is to find a compromise between Bayesian<fnref n="4" /> and Frequentist approaches, that may be considered more

flexible.

Much research has gone into finding a means of calibrating *p*-values so that they are comparable with typical Bayesian based metrics (Sellke et al., 2001; Held and Ott, 2018; Benjamin et al., 2018; Benjamin and Berger, 2019). The value of such calibrations would be to consider *p*-values from a non-binary perspective; it is not simply a case of rejecting or accepting H₀ given a strict threshold, but instead assessing the probability of accepting/rejecting the hypothesis, that can be described through various degrees and magnitudes of differences. In particular, the contributions by Benjamin and Berger (2019), as well as Colquhoun (2017), (2019), are of particular interest. The latter author suggests that *p*-values be accompanied or replaced by calibrated Bayesian approaches that consider instead the probability that a *p*-value results in a Type I statistical error, known as the False Positive Risk (FPR), which have more intuitive interpretations than simple reports of *p*-values.

The outcomes of such research have reached the following conclusions (Wasserstein and Lazar, 2016; Wasserstein et al., 2019, inter alia), that are currently overlooked by a lot of members of the non-statistical community;

* The ASA officially states as of 2019 that the term “statistically significant”, and all variants of this expression, be completely abandoned. This stems from the observation that this term has become overused to such a degree that it bears little to no scientific or statistically relevant value.

* Much more information must be considered when making any form of inference based on Frequentist based statistical testing, including more transparent reporting of all aspects of statistical tests, as well as the comparison and contrast of multiple sources of information.

* *p* < 0.05 is no longer an acceptable threshold to evaluate a statistical hypothesis, given its very high probability (28.9 %) of being a Type I statistical error (see “A proposal to move forward”). While a number of alternatives have been proposed, including Greenwald et al. (1996), as well as Benjamin and Berger (2019)’s use of *p* < 0.005, the current author has proposed in the past an alternative, *p* < 0.003 (Courtenay et al., 2021a, Courtenay, 2023b), which will be clarified in continuation.

* Considering the above three points, a shift is proposed towards a less dichotomous way of thinking when using statistics; simply rejecting or accepting a given hypothesis is an oversimplification of reality.

### 4. A proposal to move forward

As far as the corresponding author is aware, few researchers in archaeology, palaeontology, or palaeoanthropology are currently adopting, or may even be aware of, these recommendations. This is surprising considering the number of research papers that have already been published in non-statistical journals over the years discussing the importance of transparent, robust, and generalisable publishing, especially where *p*-values are involved (e.g. Yoccoz, 1991; Johnson, 1999; Steel et al., 2013; White et al., 2013; Smaldino and McElreath, 2016; Smith, 2017, 2019; Courtenay et al., 2021a, b; Muff et al., 2022; Vai-glova et al., 2023). In a lot of these cases, it is important to point out that on occasions, these studies are found in either specialised papers, or journals that researchers from other fields may not frequently read or know about. Likewise, while studies exist that promote a more critical view of purely Frequentist statistical inference in these fields, including a push towards a more Bayesian approach in some cases (Konigsberg and Frankenberg, 2013), their wide-spread use is still lacking. While all of these contributions are highly valuable, their visibility to some fields in the Quaternary sciences may be more limited, presenting one of the objectives of this short communication.

It is important to point out that regardless of how reluctant some researchers may be to accept these theories, *p* < 0.05, and other forms of reporting statistics, have to improve. While this does not mean we have

<footnote n="3">³ Frequentist statistics is a framework for statistical inference that interprets probability as a long-run frequency of events. This is one of the most common school of statistics, and is strongly characterised by all of the theoretical notions presented within this text; the use of *p*-values, and the evaluation of a hypothesis given a Null and Alternative scenario.</footnote>

<footnote n="4">⁴ Bayesian statistics are an alternative framework, that incorporates the concept of prior probability and uncertainty into calculations to calculate the probability that an event occurs, given Bayes’ theorem. The output of such computations is an updated belief on the probability that a hypothesis is true. While Bayesian statistics are employed in archaeology and palaeoanthropology, their wide-spread use is still limited.</footnote>
<caption>**Table 1**</caption>

– Table presenting the probability (sensu Colquhoun 2017, 2019) that a given *p*-value (columns) results in a Type I statistical error, given different prior probabilities (rows) in favour of the Null Hypothesis.

<table><thead><tr><th>Prior p-value</th><th>0.368</th><th>0.100</th><th>0.050</th><th>0.010</th><th>0.005</th><th>0.003</th><th>0.001</th><th>0.0001</th></tr></thead><tbody><tr><td>0.1</td><td>0.900</td><td>0.849</td><td>0.786</td><td>0.530</td><td>0.393</td><td>0.299</td><td>0.144</td><td>0.022</td></tr><tr><td>0.2</td><td>0.800</td><td>0.715</td><td>0.620</td><td>0.334</td><td>0.224</td><td>0.159</td><td>0.070</td><td>0.010</td></tr><tr><td>0.3</td><td>0.700</td><td>0.594</td><td>0.487</td><td>0.226</td><td>0.144</td><td>0.100</td><td>0.042</td><td>0.006</td></tr><tr><td>0.4</td><td>0.600</td><td>0.484</td><td>0.379</td><td>0.158</td><td>0.097</td><td>0.066</td><td>0.027</td><td>0.004</td></tr><tr><td>0.5</td><td>0.500</td><td>0.385</td><td>0.289</td><td>0.111</td><td>0.067</td><td>0.045</td><td>0.018</td><td>0.002</td></tr><tr><td>0.6</td><td>0.400</td><td>0.294</td><td>0.213</td><td>0.077</td><td>0.046</td><td>0.031</td><td>0.012</td><td>0.002</td></tr><tr><td>0.7</td><td>0.300</td><td>0.212</td><td>0.149</td><td>0.051</td><td>0.030</td><td>0.020</td><td>0.008</td><td>0.001</td></tr><tr><td>0.8</td><td>0.200</td><td>0.135</td><td>0.092</td><td>0.030</td><td>0.018</td><td>0.012</td><td>0.005</td><td>0.001</td></tr><tr><td>0.9</td><td>0.100</td><td>0.065</td><td>0.043</td><td>0.014</td><td>0.008</td><td>0.005</td><td>0.002</td><td>0.000</td></tr></tbody></table>

to completely reject all of the statistics that we know, it does imply that a revolution and a refinement of our practices is certainly required to ensure more transparent and robust hypothesis testing.

**5. From this perspective, we have to be aware of the following, and adjust our practices to at least try and adapt to these recommendations;**

* Exact *p*-values must be reported for all research, unless they are sufficiently far enough away from the mean of $H_0$ that absolutely no possibility exists of being misinterpreted (e.g., *p* < 2.2e-16, a common output of R functions). This means that tables presenting the all too common expression “< 0.01” have to be discarded.

* Authors who exclusively publish *p*-values without any of the associated test statistics need to be more transparent with reporting their results. For example; when reporting ANOVA results, reporting the *p*-value is not enough, we must include the *F* statistic and nature of the residuals as well. For correlation calculations, we have to report not just the *p*-value and coefficient, but the test statistic as well. For Shapiro-Wilk's tests, the test statistic *W* must be included as well as the *p*-value. For any test that includes a degree of freedom, it is important to be transparent about these values as well (and so on).

* Authors should be aware of the limitations of using traditional parametric statistical tests (ANOVA, $\chi^2$, etc.) and descriptive statistics ($\mu$, $\sigma$, etc.), when normality is not met, and should use non-parametric and robust alternatives instead (Courtenay, 2023b). While this could be the topic of a completely different “open letter to evolutionary and human sciences”, it is worth reinforcing here.

* It is highly recommended that *p*-values be calibrated with other Frequentist-to-Bayesian based metrics, such as the False Positive Risk, as proposed by Colquhoun (2017), (2019). Other alternatives include reporting posterior odds and Bayes Factor Bound values, as recommended by Benjamin and Berger (2019). While these calibrations require the analyst to provide a prior probability in favour of $H_a$, if the analyst is unsure and wishes to use vague priors, then this value is automatically set to 0.5 (Colquhoun, 2017, 2019). To facilitate these calculations, the corresponding author has created the pValueRobust R library, available on Github (https://github.com/LACourtenay/pValueRobust). Given the observations by Courtenay et al. (2021b), these calibrations and complementary calculations are especially relevant for all *p*-values below 0.3681

* As similarly stated by Smith (2017, 2019), reporting effect sizes and confidence intervals is highly recommended, while comparing multiple metrics and statistical tests should be considered essential to ensure consistent, replicable conclusions. No test or result should be considered in isolation, especially when multiple complementary analyses exist.

* Binary reject-or-accept $H_0$ scenarios, based on conventional thresholds such as *p* < 0.05, should be completely abandoned.

Finding solutions to this last point is not entirely straightforward, and a number of alternative approaches exist, especially with regards to

possible alternative *p*-value thresholds. The Frequentist-to-Bayesian calibration approach would consider that different magnitudes of differences can be reported (Sellke et al., 2001; Held and Ott, 2018; Muff et al., 2022), similar to how Bayes Factors are interpreted, with *p*-values between certain intervals indicating progressively larger differences. Benjamin et al. (2018) and Benjamin and Berger (2019), for example, suggest that values below *p* < 0.005 can be considered more conclusive, while values between 0.05 and 0.005 are only “suggestive” of $H_a$. In Muff et al. (2022) the authors present a more gradual shift, where values from 0.05 to 0.01 imply moderate evidence, 0.01–0.001 strong evidence, and <0.001 very strong evidence. While each of these approaches are interesting and a better solution to simply providing a dichotomous threshold, the corresponding author makes a different proposal, and argues *p* < 0.003 to be a more reasonable and epistemologically relevant definition of a threshold. This value is derived from 1 – 3$\sigma$ ≈ 0.003, and thus considers that in many cases from the field of robust statistics, 3$\sigma$ is slightly more robust than 2$\sigma$ to skewed, noisy or inhomogeneous distributions. From this perspective, if such a binary threshold between what is considered conclusive and non-conclusive were to exist, then 0.003 could be argued to be more epistemologically well-defined.

As a final push to clarify why *p* < 0.003 outperforms *p* < 0.05 as a more robust threshold for evaluating $H_0$, we can consider the probability of Type I statistical errors given the use of different *p*-values. Table 1 contains the probability of different *p*-values being a Type I statistical error, following Colquhoun (2017), (2019), while using the Selke-Berger approach to define internal parameters of the formulae (Sellke et al., 2001; Benjamin and Berger, 2019). As can be seen, rejecting the null hypothesis based on a *p*-value of 0.05 has a 28.9 % of being a False Positive, i.e. we reject a hypothesis that is in fact true.

To illustrate these concepts, we can consider the following hypothetical scenarios. If we consider the use of an ANOVA test between two sets of measurements yields a *p*-value of 0.049, traditionally, this would be taken as evidence that differences exist between the samples (*p* < 0.05), and could potentially lead some authors to draw “conclusive” remarks based on such evidence. Nevertheless, this value is far from conclusive. With a *p*-value of 0.049, there is a 28.7 % chance that this finding is actually a mistake. If we assume a prior probability of 0.2 in favour of $H_0$, this *p*-value translates to a 61.6 % probability of a statistical error. Furthermore, if we inappropriately use ANOVA on non-homogeneous samples, the likelihood of a false conclusion remains around 61.6 %.

If we were to explore this latter scenario further, yet use a more suitable test like Kruskal-Wallis resulting in a *p*-value of 0.003, and under the assumption that our hypothetical measurements are more likely to present differences, this *p*-value indicates a 4.5 % chance of error. Even with a prior probability of 0.2, the error probability is reduced to 15.9 %. These scenarios underscore the importance of adopting more rigorous statistical thresholds. By moving from *p* < 0.05 to *p* < 0.003, we enhance the reliability of our findings and advance the robustness of our scientific claims.

Needless to say, this is not to say that statistical practices should simply substitute *p* < 0.05 for *p* < 0.003. Simply defining a new
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** – The calculated minimum sample sizes as a function of statistical power, using different levels of α, for (left) a two-one sided z test, with a moderate effect size of 0.5, and (right) an ANOVA differentiating between 2 groups. Note that α is the ‘significance level’ for each of the calculations, i.e. an indication that a *p*-value less than or equal to α would result in the rejection of the null hypothesis, thus providing evidence for the effect we are trying to detect.</figcaption></figure>

threshold for *p*-value evaluations would still not consider the non-dichotomous nature of most real-life hypothesis testing scenarios, and would still not fulfil the recommendations of statisticians regarding the robusticity, repeatability, generalisability, and transparency, of reporting statistical results. In this scenario, it is here where individual statistical analyses must evaluate, reflect upon, and consider the effect size and patterns they are trying to detect. For this purpose, reporting FPR values alongside *p*-values, with suitable priors, could be a valuable means of evaluating how likely it is we are going to be making a mistake. Likewise, depending on the application, adopting a graduated series of thresholds, may be more appropriate (Sellke et al., 2001; Held and Ott, 2018; Muff et al., 2022), as long as they are well and empirically justified.

**6. Concluding remarks**

It is fundamental to point out that none of the presented concepts are new, however they have gone unnoticed by the general human sciences, published for a specialised audience (e.g., Vaiglova et al., 2023), or were published as complementary parts to other specialised studies (e.g., Courtenay et al. 2021a). Nevertheless, this does not mean that they are not important. The objective of the present communication was therefore to once again ask a wider audience to consider these issues, and recommend that analyses in the human and evolutionary sciences begin to adapt and adopt the recommendations of statisticians.

As a closing statement, it is also useful to accept that, as with any scientific discourse, there is always a counter-argument. From one side, it is completely reasonable to assume that pushing back the line in Fig. 1 will of course result in an increase of Type II statistical errors - we cannot reduce the probability of one event without considering the probability that another will occur instead. Such a notion was discussed by Benjamin et al. (2018), who additionally point out that to maintain statistical power with lower *p*-value thresholds, we would require an increase in sample sizes (Fig. 2) – something we are unlikely to fulfil in fields such as archaeology. Likewise, simply changing the *p*-value threshold only solves a small part of the issue, and many researchers can simply replace 0.05 with 0.005 or 0.003 and still report misleading and biased statistical inferences (Amrhein and Greenland, 2018).

It is fundamental to remind the reader, therefore, that analysts have a responsibility to ensure that they are testing the correct hypotheses, be transparent about them, and consider whether the gains of changing our testing methods outweigh the losses. To assess the potential increase in

Type II errors, we must be completely aware of what our $H_0$ is, and orient our testing to consider the most appropriate research questions, rather than simply applying techniques that are not well suited for this purpose. For example, if we are performing tests focusing on how similar samples are, then of course Type II errors will increase for ANOVA, as the $H_0$ is not well suited for this type of analysis (i.e. consider equiva-lency testing instead). With regards to sample sizes, this is a true concern, but can be overcome by understanding the limitations of making any type of inference on a fragmentary and incomplete fossil record anyway. Such a topic, alongside sampling strategies, has been touched upon by a number of researchers, from multiple perspectives (e. g., Cannon, 2001; Cochrane, 2003; Cardini and Elton, 2007; Haslam, 2009; Smith, 2012; Lee, 2012; Cardini et al., 2015; Barr et al., 2022; Vaiglova et al., 2023). Likewise, ensuring that the methodology we are using is as transparent, replicable and repeatable as possible will help in the interpretation, corroboration, and validation of results (Marwick, 2016). It is therefore fundamental that we not only refrain from considering *p*-values alone, but instead push towards a more non-dichotomous and more transparent means of performing, reporting, and consequently publishing our results.

**Declaration of Competing Interest**

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Lloyd A Courtenay reports financial support was provided by European Research Council. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

**Acknowledgements**

This research was funded by the European Research Council Synergy Grant for the project Evolution of Cognitive Tools for Quantification (QUANTA), No. 951388. Funders had no role in the design or writing of this study. Beyond this, the corresponding author would like to thank the feedback and discussions had with three anonymous reviewers on earlier versions of this manuscript.

**References**

Amrhein, V., Greenland, S., 2018. Remove, rather than redefine, statistical significance. Nat. Hum. Behav. 2, 4. https://doi.org/10.1038/s41562-017-0224-0.
Barr, A., Pobiner, B., Rowan, J., Du, A., Faith, J.T., 2022. No sustained increase in zooarchaeological evidence for carnivory after the appearance of *Homo erectus*. *PNAS* 119 (5), e2115540119.

Benjamin, D.J., Berger, J.O., 2019. Three recommendations for improving the use of p-values. Am. Stat. 73 (sup1), 186-191. https://doi.org/10.1080/0031305.2018.1543135.

Benjamin, D.J., Berger, J.O., Johannesson, M., Nosek, B.A., Wagenmakers, E.J., Berk, R., Bollen, K.A., Brembs, B., Brown, L., Camerer, C., Cesarini, D., Chambers, C.D., Clyde, M., Cook, T.D., Boeck, P., Dienes, Z., Dreber, A., Easwaran, L., Efferson, C., Fehr, E., Fidler, F., Field, A.P., Forster, M., George, E.I., Gonzalez, R., Goodman, S., Green, E., Green, D.P., Greenwald, A.G., Hadfield, J.D., Hedges, L.V., Held, L., Ho, T. H., Hruschka, D.J., Imai, K., Imbens, G., Ioannidis, J.P.A., Jeon, M., Jones, J.H., Kirchler, M., Laibson, D., List, J., Little, R., Lupia, A., Machinery, E., Maxwell, S.E., McCarthy, M., Moore, D.A., Morgan, S.L., Munafò, M., Nakagawa, S., Nyhan, B., Parker, T.H., Pericicchi, L., Perugini, M., Rouder, J., Rousseau, J., Savalei, V., Schönbrodt, F.D., Sellke, T., Sinclair, B., Tingley, D., Van Zandt, T., Vazire, S., Watts, D.J., Winship, C., Wolpert, R.L., Xie, Y., Young, C., Zinman, J., Johnson, V.E., 2018. Redefine statistical significance. Nat. Hum. Behav. 2, 6-10. https://doi.org/10.1038/s41562-017-0189-z.

Cannon, M.D., 2001. Archaeofaunal relative abundance, sample size, and statistical methods. J. Archaeol. Sci. 28 (2), 185-195.

Cardini, A., Elton, S., 2007. Sample size and sampling error in geometric morphometric studies of size and shape. Zoomorphology 126, 121-134.

Cardini, A., Seetah, K., Barker, G., 2015. How many specimens do I need? Sampling error in geometric morphometrics: testing the sensitivity of means and variances in simple randomized selection experiments. Zoomorphology 134, 149-163.

Cochrane, G.W.G., 2003. Artefact attribute richness and sample size adequacy. J. Archaeol. Sci. 30 (7), 837-848.

Cohen, J., 1994. The earth is round (*p* < .05). Am. Psychol. 49 (12), 997-1003.

Colquhoun, D., 2017. The reproducibility of research and the misinterpretation of p-values. R. Soc. Open Sci. 4, 171085. https://doi.org/10.1098/rsos.171085.

Colquhoun, D., 2019. The False Positive Risk: a proposal concerning what to do about p-values. Am. Stat. 73 (Sup1), 192-201. https://doi.org/10.1080/00031305.2018.1529622.

Courtenay, L.A., 2023a. Exploring the reliability of handaxe morphological analyses in 2D: a simulation-based approach. Archaeol. Anthropol. Sci. 15, 102. https://doi.org/10.1007/s1252-023-01807-3.

Courtenay, L.A., 2023b. Robust statistical and artificially intelligent approaches for the analysis of 2D and 3D morphological data. PhD Thesis. University of Salamanca.

Courtenay, L.A., González-Aguilera, D., Lagiuela, S., del Pozo, S., Ruiz-Mendez, C., Barbero-García, I., Román-Curto, C., Cañueto, J., Santos-Durán, C., Cardenoso-Álvarez, M.E., Roncero-Riesco, M., Hernandez-Lopez, D., Guerrero-Sevilla, D., Rodríguez-Gonzalvez, P., 2021a. Hyperspectral imaging and robust statistics in non-melanoma skin cancer analysis. Biomed. Opt. Express 12, 5107-5127. https://doi.org/10.1364/BOE.428143.

Courtenay, L.A., Herranz-Rodrigo, D., González-Aguilera, D., Yravedra, J., 2021b. Developments in data science solutions for carnivore tooth pit classification. Sci. Rep. 11, 10209. https://doi.org/10.1038/s41598-021-89518-4.

Edgeworth, M.A., 1885. Methods of statistics. J. Stat. Soc. 41, 181-217.

Elderton, W.P., 1902. Tables for testing the goodness of fit of theory to observation. Biometrika 1, 155-163.

Feyerabend, P., 1993. Against Method. Verso, Vienna.

Fisher, R.A., 1925. Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh.

Goodman, S., 2008. A dirty dozen: twelve *p*-value misconceptions. Semin. Hematol. 45, 135-140. https://doi.org/10.1053/j.seminhematol.2008.04.003.

Greenwald, A.G., Gonzalez, R., Harris, R.J., Guthrie, D., 1996. Effect sizes and *p* values: what should be reported and what should be replicated? Psychophysiology 33 (2), 175-183.

Haslam, M., 2009. Mountains and molehills: sample size in archaeological microscopic sone-tool residue analysis. In: Haslam, M., Robertson, G., Crowther, A., Nugent, S.,

Kirkwood, L. (Eds.), Archaeological Science Under a Microscope. Australian National University, Canberra, pp. 47-79.

Held, L., Ott, M., 2018. On *p*-values and Bayes factors. Annu. Rev. Stat. Appl. 5 (6)), 1-27.

Johnson, D.H., 1999. The insignificance of statistical significance testing. J. Wildl. Manag. 63, 763-772.

Kennedy-Schaffer, L., 2019. Before *p*<0.05 to beyond *p*<0.05: using history to contextualize *p*-values and significance testing. Am. Stat. 73 (sup1), 82-90. https://doi.org/10.1080/00031305.2018.1537891.

Konigsberg, L.W., Frankenberg, S.R., 2013. Bayes in biological anthropology. Am. J. Phys. Anthropol. 152, 153-184.

Kuhn, T., 1962. The Structure of Scientific Revolutions. University of Chicago Press, Chicago.

Laplace, P.S., 1823. Théorie Analytique des Probabilités. Courrier, Paris.

Lee, G.A., 2012. Taphonomy and sample size estimation in paleoethnobotany. J. Archaeol. Sci. 39 (3), 648-655.

Marwick, B., 2016. Computational reproducibility in archaeological research: basic principles and a case study of their implementation. J. Archaeol. Method Theory 24, 424-450.

Marwick, B., Boettiger, C., Mullen, L., 2018. Packaging data analytical work reproducibly using R (and friends). Am. Stat. 72 (1), 80-88.

Muff, S., Nilsen, E.B., O'Hara, R.B., Nater, C.R., 2022. Rewriting results sections in the language of evidence. Trends Ecol. Evol. 37 (3), 203-210.

Neyman, J., Pearson, E.S., 1933a. The testing of statistical hypotheses in relation to probabilities a priori. Math. Proc. Camb. Philos. Soc. 29 (4), 492-510.

Neyman, J., Pearson, E.S., 1933b. On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. A 231, 289-337.

Pearson, K., 1895. On the criterion that a given system of deviations from the portable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Lond. Edinb. Dublin Philos. Mag. J. Sci. 50 (302), 151-175.

Poisson, S.D., 1837. Recherches sur la probabilité des jugements en matière criminelle et en matière civile: précédées de règles générales du calcul des probabilités. Bachelier, Paris.

Sellke, T., Bayarri, M.J., Berger, J.O., 2001. Calibration of *p* values for testing precise null hypotheses. Am. Stat. 55 (1), 62-71.

Smaldino, P.E., McElreath, R., 2016. The natural selection of bad science. R. Soc. Open Sci. 3, 160384. https://doi.org/10.1098/rsos.160384.

Smith, M.E., 2012. The Comparative Archaeology of Complex Societies. Cambridge University Press, Cambridge.

Smith, R.J., 2017. The continuing misuse of null hypothesis significance testing in biological anthropology. Am. J. Phys. Anthropol. 166 (1), 236-245.

Smith, R.J., 2019. *P* > .05: The incorrect interpretation of “not significant” results is a significant problem. Am. J. Phys. Anthropol. 172 (4), 521-527.

Steel, E.A., Kennedy, M.C., Cunningham, P.G., Stanovick, J.S., 2013. Applied statistics in ecology: common pitfalls and simple solutions. Ecosphere 4 (9), 115. https://doi.org/10.1890/ES13-00160.1.

Student, 1908. The probable error of a mean. Biometrika 6, 1-25.

Vaiglova, P., Lazar, N.A., Stroud, E.A., Loftus, E., Makarewicz, C.A., 2023. Best practices for selecting samples, analysing data, and publishing results in isotope archaeology. Quat. Int. 650, 86-100.

Wasserstein, R.L., Lazar, N.A., 2016. The ASA statement on *p*-values. Context, process and purpose. Am. Stat. 70 (2), 129-133. https://doi.org/10.1080/0031305.2016.1154108.

Wasserstein, R.L., Schirm, A.L., Lazar, N.A., 2019. Moving to a world beyond "*p* < 0.05". Am. Stat. 73 (Sup1), 1-19.

White, J.W., Rassweiler, A., Samhouri, J.F., Stier, A.C., White, C., 2013. Ecologists should not use statistical significance tests to interpret simulation model results. Oikos 123 (4), 385-388. https://doi.org/10.1111/j.1600-0706.2013.01073.x.

Yoccoz, N.G., 1991. Use, overuse, and misuse of significance tests in evolutionary biology and ecology. Bull. Ecol. Soc. Am. 27, 106-111.