SlowGuess/ACL_2020_OCR huggingface dataset:
sis H3 (with a margin of  $0\%$ ) holds true. As explained in §C.2, this statement does not imply any difference with a notable margin. In fact, the posterior in Figure 2 implies that this experiment is not sufficient to claim the following: with probability at least 0.95, hypothesis H3 (with a margin of  $1\%$ ) holds true. This is the case since ROPE (0.01, 0.01) overlaps with  $95\%$  HDI (0.00939, 0.0612).
2.4 Bayes Factor
A common tool among Bayesian frameworks is the notion of Bayes Factor. Intuitively, it compares how the observations  $\mathbf{y}$  shift the credibility from prior to posterior of the two competing hypothesis:
$$
B F _ {0 1} = \frac {\mathbb {P} (H _ {0} | \boldsymbol {y})}{\mathbb {P} (H _ {1} | \boldsymbol {y})} / \frac {\mathbb {P} (H _ {0})}{\mathbb {P} (H _ {1})}
$$
If the  $BF_{01}$  equals to 1 then the data provide equal support for the two hypotheses and there is no reason to change our a priori opinion about the relative likelihood of the two hypotheses. A smaller Bayes Factor is an indication of rejecting the null-hypothesis  $H_0$ . If it is greater than 1 then there is support for the null-hypothesis and we should infer that the odds are in favor of  $H_0$ .
Notice that the symmetric nature of Bayes Factor allows all the three outcomes of "accept", "reject", and "undecided," as opposed to the definition of  $p$ -value that cannot accept a hypothesis.
Example 4 (Assessment of H4) Here we want to assess the null-hypothesis  $H_0$ :  $|\theta_1 - \theta_2| < 0.01$  against  $H_1$ :  $|\theta_1 - \theta_2| \geq 0.01$  ( $x = 0.01$ ). Substituting posterior and prior values, one obtains:
$$
B F _ {0 1} = \frac {0 . 0 2 7}{0 . 9 8 0} / \frac {0 . 0 1 9}{0 . 9 7 2} = 1. 3 8 2
$$
. This value is very close to 1 which means that this observation does not change our prior belief about the two systems difference.
3 Comparisons
Many aspects influence the choice of an approach to assess significance of hypotheses. This section provides a comparative summary, with details in Appendix C and an overall summary in Table 3.
3.1 Susceptibility to Misinterpretation
The complexity of interpreting significance tests combined with insufficient reporting could result in ambiguous or misleading conclusions. This ambiguity can not only confuse authors but also cause confusion among readers of the papers.
While  $p$ -values (§2.1) are the most common approach, they are inherently complex, which makes them easier to misinterpret (see examples in §C.1). Interpretation of confidence intervals (§2.2) can also be challenging since it is an extension of  $p$ -value (Hoekstra et al., 2014). Approaches that provide measures of uncertainty directly in the hypothesis space (like the ones in §2.3) are often more
9 "Bayesian Hypothesis Testing" usually refers to the arguments based on "Bayes Factor." However, as shown in §2.3, there are other Bayesian approaches for assessing hypotheses.
Method	Paradigm	Ease of interpretation (1 = easy) (§3.1)	Encourages binary-thinking (3.2)	Depends on stopping intention (3.3)	Dependence on prior (3.4)	Decision rule	# of papers using this test in ACL'18
(§2.1)p-value	frequentist	3	Yes	Yes	No	Acceptable p-value	73
(§2.2) CI	frequentist	4	No	Yes	No	Acceptable confidence margin	6
(§2.3) HDI	Bayesian	1	No	No	Not sensitive but takes it into account	HDI relative to ROPE	0
(§2.4) BF	Bayesian	2	Yes	No	Highly sensitive	Acceptable BF	0
Table 3: A comparison of different statistical methods for evaluating the credibility of a hypothesis given a set of observations. The total number of published papers in at the ACL-2018 conference is 439.
natural choices for reporting the results of experiments (Kruschke and Liddell, 2018).
3.2 Measures of Certainty
A key difference is that not all methods studied here provide a measure of uncertainty over the hypothesis space. For instance,  $p$ -values (§2.1) do not provide probability estimates on two systems being different (or equal) (Goodman, 2008). On the contrary, they encourage binary thinking (Gelman, 2013), that is, confidently concluding that one system is better than another, without taking into account the extent of the difference between the systems. CIs (§2.2) provide a range of values for the target parameter. However, this range also does not have any probabilistic interpretation in the hypothesis space (du Prel et al., 2009). On the other hand, posterior intervals (§2.3) generally provide a useful summary as they capture probabilistic estimates of the correctness of the hypothesis.
3.3 Dependence on Stopping Intention
The process by which samples in the test are collected can affect the outcome of a test. For instance, the sample size  $n$  (whether it is determined before the process of gathering information begins, or is a random variable itself) can change the result. Once observations are recorded, this distinction is usually ignored. Hence, the testing algorithms that do not depend on the distribution of  $n$  are more desirable. Unfortunately, the definition of  $p$ -value (\$2.1) depends on the distribution of  $n$ . For instance, Kruschke (2010, §11.1) provides examples where this subtlety can change the outcome of a test, even when the final set of observations is identical.
3.4 Sensitivity to the Choice of Prior
The choice of the prior can change the outcome of Bayesian approaches (§2.3 & §2.4). Decisions of Bayes Factor (§2.4) are known to be sensitive to
the choice of prior, while posterior estimates (§2.3) are less so. For further discussion, see C.4 or refer to discussions by Sinharay and Stern (2002); Liu and Aitkin (2008) or Dienes (2008).
4 Current Trends and Malpractices
This section highlights common practices relevant to the our target approaches. To better understand the common practices or misinterpretations in the field, we conducted a survey. We shared the survey among  $\sim 450$  NLP researchers (randomly selected from ACL'18 Proceedings) from which 55 individuals filled out the survey. While similar surveys have been performed in other fields (Windish et al., 2007), this is the first in the NLP community, to the best of our knowledge. Here we review the main highlights (see Appendix for more details and charts).
Interpreting  $p$ -values. While the majority of the participants have a self-claimed ability to interpret  $p$ -values (Figure 9f), many choose its imprecise interpretation "The probability of the observation this extreme happening due to pure chance" (the popular choice) vs. a more precise statement "Conditioned on the null hypothesis, the probability of the observation this extreme happening." (see Q1 & Q2 in Appendix B.)
The use of CIs. Even though  $95\%$  percent of the participants self-claimed the knowledge of CIs (Figure 9e), it is rarely used in practice. In an annotation done on ACL'18 papers by two of the authors, only 6 (out of 439) papers were found to use CIs.
The use of Bayes Factors. A majority of the participants had "heard" about "Bayesian Hypothesis Testing" but did not know the definition of "Bayes Factor" (Figure 3). HDIs (discussed in §2.3) were the least known. We did not find any papers in ACL'18 that use Bayesian tools.
5721
Do you know the definition of "Bayes Factor"?
Figure 3: Select results from our survey.
Have you heard about "Bayesian Hypothesis Testing"?
Do you know the definition of "Highest Density Interval"?
The use of "significant*". A notable portion of NLP papers express their findings by using the term "significant" (e.g., "our approach significantly improves over  $X$ .") Almost all ACL'18 papers use the term "significant"10 somewhere. Unfortunately, there is no single universal interpretation of such phrases across readers. In our survey, we observe that when participants read "X significantly improves Y" in the abstract of a hypothetical paper:
1. About  $82\%$  expect the claim to be backed by "hypothesis testing"; however, only  $57\%$  expect notable empirical improvement (see Q3 in Appendix B);
2. About  $35\%$  expect the paper to test "practical significance", which is not generally assessed by popular tests (see  $\S \mathbf{C}.2$ );
3. A few also expect a theoretical argument.
Recent trends. Table 3 provides a summary of the techniques studied here. We make two key observations: (i) many papers don't use any hypothesis assessment method and would benefit from one; (ii) from the final column,  $p$ -value based techniques clearly dominate the field, a clear disregard to the advantages that the bottom two alternatives offer.
5 Recommended Practices
Having discussed common issues, we provide a collection of recommendations (in addition to the prior recommendations, such as by Dror et al. (2018)).
The first step is to define your goal. Each of the tools in §2 provides a distinct set of information. Therefore, one needs to formalize a hypothesis and consequently the question you intend to answer by assessing this hypothesis. Here are four representative questions, one for each method:
1. Assuming that the null-hypothesis is true, is it likely to witness observations this extreme? (§2.1)
2. How much my null-hypothesis can deviate from the mean of the observations until a  $p$ -value argument rejects it. (§2.2)
3. Having observed the observations, how probable is my claimed hypothesis?( $\S 2.3$ )
4. By observing the data how much do the odds increase in favor of the hypothesis?( $\S 2.4$ )
If you decide to use frequentist tests:
- Check if your setting is compatible with the assumptions of the test. In particular, investigate if the meaning of null-hypothesis and sampling distribution match the experimental setting.
- Include a summary of the above investigation. Justify unresolved assumption mismatches.
- Statements reporting  $p$ -value and confidence interval must be precise enough so that the results are not misinterpreted (see §3.1).
- The term "significant" should be used with caution and clear purpose to avoid misinterpretations (see §4). One way to achieve this is by using adjectives "statistical" or "practical" before any (possibly inflected) usage of "significance."
- Often times, the desired conclusion is a notable margin in the superiority of one system over another (see §3). In such cases, a pointwise  $p$ -value argument is not sufficient; a confidence interval analysis is needed. If CI is inapplicable for some reason, this should be mentioned.
If you decide to use Bayesian approaches:
- Since Bayesian tests are less known, it is better to provide a short motivation for the usage.
- Familiarize yourself with packages that help you decide a hierarchical model, e.g., the software provided here. If necessary, customize these models for your specific problem.
- Be clear about your hierarchical model, including model parameters and priors. In most cases, these choices should be justified (see §2.3.)
10Or other variants "significantly", "significance", etc.
5722
Statistical Model	Observation Type	Hierarchical Model	Assumptions	Parameters	Common settings / metrics	Common Frequentist test (Parametric)	Common Frequentist test (Non-Parametric)
Binary model	binary output	Bernoulli distribution with Beta prior	2	For each group: p ∈ [0,1] (success probability)	correct vs incorrect predictions	Binomial test	bootstrap / permutation
Binomial model	binomial output	Binomial distribution with Beta prior	2,3,6	For each group: p ∈ [0,1] (success probability)	Exact match, Accuracy, Recall, UAS (sentencelevel), LAS (sentencelevel)*	Binomial test	bootstrap / permutation
Metric model	metric observations	T-Student distribution with multiple priors *	1,2,4	For each group: mu ∈ R and sigma ∈ R+ Shared between groups: nu ∈ R+ (normality parameter)	Exact match, Accuracy, Recall, UAS (sentencelevel), LAS (sentencelevel), running time. energy usage, L2 error	t-test	bootstrap / permutation
Count model	counts	Negative Binomial distribution with Normal prior	2,5	For each group: mu ∈ R+ (rate parameter) alpha ∈ R+ (shape parameter)	The count of certain patterns an algorithm could find in a big pool, in a fixed amount of time. Notice that you can't convert this into a ratio form, since there is no well-defined denominator. Ex: measuring how many of questions could be answered correctly (from an infinite pool of questions) by a particular QA systems, in a limited minute (the system is allowed to skip the questions too)	bootstrap / permutation
Ordinal model	ordinals	Normal distribution with parameterized thresholds	2	For each group: mu ∈ R and sigma ∈ R+ Shared between groups: thresholds between possible levels	Collection of objects/labels arranged in a certain ordering, not necessarily with a metric distance between them; for example sentiment labels (https://www.acweb.org/anthology/S16-1001.pdf), product review categories, grammaticality of sentences	bootstrap / permutation
Assumption 1: The observations are distributed as a t-student with unknown normality parameter (a normal distribution with potentially longer tales).
Assumption 2: The observations from each group are assumed to be i.i.d, conditioned on the inherent characteristics of two systems
Assumption 3: The total number of instances (the denominators) is known.
Assumption 4: The variable is inherently continuous, or the granularity (the denominator) is high enough to treat the variable as continuous.
Assumption 5: The observations follow a Negative-Binomial / Poisson distribution.
Assumption 6: The observations follow a binomial-distribution.
Assumption 7: The observations follow a normal distribution.
\* In this model (unlike frequentist t-test) outliers don't need to be discarded manually to realize the strict normality assumption.
Table 4: Select models supported by our package HyBayes at the time of this publication.
- Comment on the certainty (or the lack of) of your inference in terms of HDI and ROPE: (I) is HDI completely inside ROPE, (II) they are completely disjoint, (III) HDI contains values both inside and outside ROPE (see §2.3.)
- For reproducibility, include further details about your test: MCMC traces, convergence plots, etc. (Our HyBayes package provides all of this.)
- Be wary that Bayes Factor is highly sensitive to the choice of prior (see §3.4). See Appendix §C.4 for possible ways to mitigate this.
5.1 Package HyBayes
We provide an accompanying package, HyBayes, to facilitate comparing systems using the two Bayesian hypothesis assessment approaches discussed earlier: (a) posterior probabilities and (b) Bayes Factors. (Several packages are already available for frequentist assessments.)
Table 4 summarizes common settings in which HyBayes can be employed $^{11}$  in NLP research, including typical use cases, underlying data assumptions, recommended hierarchical model, metrics (accuracy, exact match, etc.), and frequentist tests generally used in these cases. These settings cover several typical assumptions on observed NLP data. However, if a user has specific information on observations or can capitalize on other assumptions, we recommend adding a custom model, which can be done relatively easily.
Using well-founded mechanisms for assessing the validity of hypotheses is crucial for any field that relies on empirical work. Our survey indicates that the NLP community is not fully utilizing scientific methods geared towards such assessment, with only a relatively small number of papers using such methods, and most of them relying on  $p$ -value.
Our goal was to review different alternatives, especially a few often ignored in NLP. We surfaced various issues and potential dangers of careless use and interpretations of different approaches. We do not recommend a particular approach. Every technique has its own weaknesses. Hence, a researcher should pick the right approach according to their needs and intentions, with a proper understanding of the techniques. Incorrect use of any technique can result in misleading conclusions.
We contribute a new toolkit, HyBayes, to make it easy for NLP practitioners to use Bayesian assessment in their efforts. We hope that this work provides a complementary picture of hypothesis assessment techniques for the field and encourages more rigorous reporting trends.
The authors would like to thank Rotem Dror, Jordan Kodner, and John Kruschke for invaluable feedback on an early version of this draft. This work was partly supported by a gift from the Allen Institute for AI and by DARPA contracts FA8750-19-2-1004 and FA8750-19-2-0201.
These settings are available at the time of this publication, with more options likely to be added in the future.
5723
Valentin Amrhein, Franzi Korner-Nievergelt, and Tobias Roth. 2017. The earth is flat  $(\mathrm{p} > 0.05)$ : significance thresholds and the crisis of unreplicable research. PeerJ, 5:e3544.
Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of EMNLP, pages 995-1005.
James O Berger and Thomas Sellke. 1987. Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82(397):112-122.
Nancy Chinchor. 1992. The statistical significance of the MUC-4 results. In Proceedings of the 4th conference on Message understanding, pages 30-50. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457.
Janez Demšar. 2008. On the appropriateness of statistical tests in machine learning. In Workshop on Evaluation Methods for Machine Learning in conjunction with ICML, page 65.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171-4186.
Zoltan Dienes. 2008. Understanding psychology as a science: An introduction to scientific and statistical inference. Macmillan International Higher Education.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhikers guide to testing statistical significance in natural language processing. In Proceedings of ACL, pages 1383-1392.
Rotem Dror and Roi Reichart. 2018. Recommended statistical significance tests for NLP tasks. arXiv preprint arXiv:1809.01448.
Stefan Evert. 2004. Significance tests for the evaluation of ranking methods. In Proceedings of COLING.
Dani Gamerman and Hedibert F Lopes. 2006. Markov chain Monte Carlo: stochastic simulation for Bayesian inference. CRC Press.
Andrew Gelman. 2013. The problem with p-values is how they're used.
Steven Goodman. 2008. A dirty dozen: Twelve p-value misconceptions. Seminars in Hematology, 45(3):135-140.
Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266-274.
Rink Hoekstra, Richard D Morey, Jeffrey N Rouder, and Eric-Jan Wagenmakers. 2014. Robust misinterpretation of confidence intervals. Psychonomic bulletin & review, 21(5):1157-1164.
Jeehyoung Kim and Heejung Bang. 2016. Three common misuses of p values. Dental hypotheses, 7(3):73.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP.
John K Kruschke. 2010. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5):658-676.
John K Kruschke. 2018. Rejecting or accepting parameter values in bayesian estimation. Advances in Methods and Practices in Psychological Science, 1(2):270-280.
John K Kruschke and Torrin M Liddell. 2018. The bayesian new statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a bayesian perspective. Psychonomic Bulletin & Review, 25(1):178-206.
Charles C Liu and Murray Aitkin. 2008. Bayes factors: Prior sensitivity and model generalizability. Journal of Mathematical Psychology, 52(6):362-375.
N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, and E. Teller. 1953. Equation of state calculations by fast computing machines. The journal of chemical physics, 21:1087.
Markus Ojala and Gemma C Garriga. 2010. Permutation tests for studying classifier performance. JMLR, 11(Jun):1833-1863.
Travis E Oliphant. 2006. A Bayesian perspective on estimating mean, variance, and standard-deviation from data. Technical report, Brigham Young University. https://scholarsarchive.byu.edu/facpub/278/.
Jean-Baptist du Prel, Gerhard Hommel, Bernd Röhrig, and Maria Blettner. 2009. Confidence interval or p-value?: Part 4 of a series on evaluation of scientific publications. *Deutsches Ärzteblatt International*, 106(19):335.
Stefan Riezler and John T Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 57-64.
Sandip Sinharay and Hal S Stern. 2002. On the sensitivity of Bayes factors to the prior distributions. The American Statistician, 56(3):196-201.
Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Héctor Martínez Alonso. 2014. What's in a p-value in NLP? In Proceedings of CoNLL, pages 1-10.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2018. Improving machine reading comprehension with general reading strategies. In Proceedings of NAACL.
David Trafimow and Michael Marks. 2015. Editorial. Basic and Applied Social Psychology, 37(1):1-2.
Ronald L Wasserstein, Nicole A Lazar, et al. 2016. The ASAs statement on p-values: context, process, and purpose. The American Statistician, 70(2):129-133.
Donna M Windish, Stephen J Huot, and Michael L Green. 2007. Medicine residents' understanding of the biostatistics and results in the medical literature. Jama, 298(9):1010-1022.
5725