SlowGuess/ACL_2020_OCR huggingface dataset:
Erfan Sadeqi Azer $^{1}$  Daniel Khashabi $^{2*}$  Ashish Sabharwal $^{2}$  Dan Roth $^{3}$
<sup>1</sup>Indiana University <sup>2</sup>Allen Institute for Artificial Intelligence <sup>3</sup>University of Pennsylvania
esadeqia@indiana.edu {danielk,ashishs}@allenai.org danroth@cis.upenn.edu
Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on  $p$ -value computation, which suffers from several known issues. While alternative proposals have been well debated and adopted in other fields, they remain rarely discussed or used within the NLP community. We address this gap by contrasting various hypothesis assessment techniques, especially those not commonly used in the field (such as evaluations based on Bayesian inference). Since these statistical techniques differ in the hypotheses they can support, we argue that practitioners should first decide their target hypothesis before choosing an assessment method. This is crucial because common fallacies, misconceptions, and misinterpretation surrounding hypothesis assessment methods often stem from a discrepancy between what one would like to claim versus what the method used actually assesses. Our survey reveals that these issues are omnipresent in the NLP research community. As a step forward, we provide best practices and guidelines tailored towards NLP research, as well as an easy-to-use package called HyBayes for Bayesian assessment of hypotheses, $^{1}$  complementing existing tools.
Empirical fields, such as Natural Language Processing (NLP), must follow scientific principles for assessing hypotheses and drawing conclusions from experiments. For instance, suppose we come across the results in Table 1, summarizing the accuracy of two question-answering (QA) systems  $S_{1}$  and  $S_{2}$  on some datasets. What is the correct way to interpret this empirical observation in terms of
System ID	Description	ARC-easy #Correct	Acc.	ARC-challenge #Correct	Acc.
S1	BERT	1721	72.4	566	48.3
S2	Reading Strategies	1637	68.9	496	42.3
Table 1: Performance of two systems (Devlin et al., 2019; Sun et al., 2018) on the ARC question-answering dataset (Clark et al., 2018). ARC-easy & ARC-challenge have 2376 & 1172 instances, respectively. Acc.: accuracy as a percentage.
the superiority of one system over another? While  $S_{1}$  has higher accuracy than  $S_{2}$  in both cases, the gap is moderate and the datasets are of limited size. Can this apparent difference in performance be explained simply by random chance, or do we have sufficient evidence to conclude that  $S_{1}$  is in fact inherently different (in particular, inherently stronger) than  $S_{2}$  on these datasets? If the latter, can we quantify this gap in inherent strength while accounting for random fluctuation?
Such fundamental questions arise in one form or another in every empirical NLP effort. Researchers often wish to draw conclusions such as:
(Ca) I'm  $95\%$  confident that  $S_{1}$  and  $S_{2}$  are inherently different, in the sense that if they were inherently identical, it would be highly unlikely to witness the observed  $3.5\%$  empirical gap for ARC-easy.
(Cb) With probability at least  $95\%$ , the inherent accuracy of  $S_{1}$  exceeds that of  $S_{2}$  by at least  $1\%$  for ARC-easy.
These two conclusions differ in two respects. First,  $\mathbf{Ca}$  claims the two systems are inherently different, while  $\mathbf{Cb}$  goes further to claim a margin of at least  $1\%$  between their inherent accuracies. The second, more subtle difference lies in the interpretation of the  $95\%$  figure: the  $95\%$  confidence expressed in  $\mathbf{Ca}$  is in terms of the space of empirical observations we could have made, given some underlying truth about how the inherent accuracies of  $S_{1}$  and  $S_{2}$  relate; while the  $95\%$  probability expressed in  $\mathbf{Cb}$  is directly over the space of possible
*Work done while the second author was affiliated with the University of Pennsylvania.
<sup>1</sup>https://github.com/allenai/HyBayes
5715
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5715-5725 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
inherent accuracies of the two systems.
To support such a claim, one must turn it into a proper mathematical statement that can be validated using a statistical calculation. This in turn brings in additional choices: we can make at least four statistically distinct hypotheses here, each supported by a different statistical evaluation:
(H1) Assuming  $S_{1}$  and  $S_{2}$  have inherently identical accuracy, the probability (p-value) of making a hypothetical observation with an accuracy gap at least as large as the empirical observation (here, 3.5%) is at most 5% (making us 95% confident that the above assumption is false).
(H2) Assuming  $S_{1}$  and  $S_{2}$  have inherently identical accuracy, the empirical accuracy gap (here, 3.5%) is larger than the maximum possible gap (confidence interval) that could hypothetically be observed with a probability of over 5% (making us 95% confident that the above assumption is false).
(H3) Assume a prior belief (a probability distribution) w.r.t. the inherent accuracy of typical systems. Given the empirically observed accuracies, the probability (posterior interval) that the inherent accuracy of  $S_{1}$  exceeds that of  $S_{2}$  by a margin of  $1\%$  is at least  $95\%$ .
(H4) Assume a prior belief (a probability distribution) w.r.t. the inherent accuracies of typical systems. Given the empirically observed accuracies, the odds increase by a factor of 1.32 (Bayes factor) in favor of the hypothesis that the inherent accuracy of  $S_{1}$  exceeds that of  $S_{2}$  by a margin of  $1\%$ .
As this illustrates, there are multiple ways to formulate empirical hypotheses and support empirical claims. Since each hypothesis starts with a different assumption and makes a (mathematically) different claim, it can only be tested with a certain set of statistical methods. Therefore, NLP practitioners ought to define their target hypothesis before choosing an assessment method.
The most common statistical methodology used in NLP is null-hypothesis significance testing (NHST) which uses  $p$ -values (Søgaard et al., 2014; Koehn, 2004; Dror and Reichart, 2018). Hypotheses H1 & H2 can be tested with  $p$ -value-based methods, which include confidence intervals and operate over the probability space of observations² ( $\S 2.1$  and  $\S 2.2$ ). On the other hand, there are often overlooked approaches, based on Bayesian inference (Kruschke and Liddell, 2018), that can be used to assess hypotheses H3 & H4 ( $\S 2.3$  and  $\S 2.4$ ) and have two broad strengths: they can deal more naturally with accuracy margins and they operate directly over the probability space of inherent accuracy (rather than of observations).
For each technique reviewed in this work, we
discuss how it compares with alternatives and summarize common misinterpretations surrounding it (§3). For example, a common misconception about  $p$ -value is that it represents a probability of the validity of a hypothesis. While desirable,  $p$ -values in fact do not provide such a probabilistic interpretation (§3.2). It is instead through a Bayesian analysis of the posterior distribution of the test statistic (inherent accuracy in the earlier example) that one can make claims about the probability space of that statistic, such as H3.
We quantify and demonstrate related common malpractices in the field through a manual annotation of 439 ACL-2018 conference papers, and a survey filled out by 55 NLP researchers (§4). We highlight surprising findings from the survey, such as the following: While  $86\%$  expressed fair-to-complete confidence in the interpretation of  $p$ -values, only a small percentage of them correctly answered a basic  $p$ -value interpretation question.
Contributions. This work seeks to inform the NLP community about crucial distinctions between various statistical hypotheses and their corresponding assessment methods, helping move the community towards well-substantiated empirical claims and conclusions. Our exposition covers a broader range of methods (§2) than those included in recent related efforts (§1.1), and highlights that these methods achieve different goals. Our surveys of NLP researchers reveals problematic trends (§4), emphasizing the need for increased scrutiny and clarity. We conclude by suggesting guidelines for better testing (§5), as well as providing a toolkit called HyBayes (cf. Footnote 1) tailored towards commonly used NLP metrics. We hope this work will encourage an improved understanding of statistical assessment methods and effective reporting practices with measures of uncertainty.
While there is an abundant discussion of significance testing in other fields, only a handful of NLP efforts address it. For instance, Chinchor (1992) defined the principles of using hypothesis testing in the context of NLP problems. Most notably, there are works studying various randomized tests (Koehn, 2004; Ojala and Garriga, 2010; Graham et al., 2014), or metric-specific tests (Evert, 2004). More recently, Dror et al. (2018) and Dror and Reichart (2018) provide a thorough review of
2More precisely, over the probability space of an aggregation function over observations, called test statistics.
<sup>3</sup>https://www.aclweb.org/anthology/events/acl-2018/
5716
frequentist tests. While an important step in better informing the community, it covers a subset of statistical tools. Our work complements this effort by pointing out alternative tests.
With increasing over-reliance on certain hypothesis testing techniques, there are growing troubling trends of misuse or misinterpretation of such techniques (Goodman, 2008; Demšar, 2008). Some communities, such as statistics and psychology, even have published guidelines and restrictions on the use of  $p$ -values (Trafimow and Marks, 2015; Wasserstein et al., 2016). In parallel, some authors have advocated for using alternate paradigms such as Bayesian evaluations (Kruschke, 2010).
NLP is arguably an equally empirical field, yet with a rare discussion of proper practices of scientific testing, common pitfalls, and various alternatives. In particular, while limitations of  $p$ -values are heavily discussed in statistics and psychology, only a few NLP efforts approach them: over-estimation of significance by model-based tests (Riezler and Maxwell, 2005), lack of independence assumption in practice (Berg-Kirkpatrick et al., 2012), and sensitivity to the choice of the significance level (Søgaard et al., 2014). Our goal is to provide a unifying view of the pitfalls and best practices, and equip NLP researchers with Bayesian hypothesis assessment approaches as an important alternative tool in their toolkit.
2 Assessment of Hypotheses
We often wish to draw qualitative inferences based on the outcome of experiments (for example, inferring the relative inherent performance of systems). To do so, we usually formulate a hypothesis that can be assessed through some analysis.
Suppose we want to compare two systems on a dataset of instances  $\boldsymbol{x} = [x_1, \ldots, x_n]$  with respect to a measure  $\mathcal{M}(S, x)$  representing the performance of a system  $S$  on an instance  $x$ . Let  $\mathcal{M}(S, x)$  denote the vector  $[\mathcal{M}(S, x_i)]_{i=1}^n$ . Given systems  $S_1, S_2$ , define  $\boldsymbol{y} \triangleq [\mathcal{M}(S_1, x), \mathcal{M}(S_2, x)]$  as a vector of observations.
In a typical NLP experiment, the goal is to infer some inherent and unknown properties of systems. To this end, a practitioner assumes a probability distribution on the observations  $\mathbf{y}$ , parameterized
Figure 1: Progression of steps taken during a scientific assessment of claims from empirical observations.
by  $\theta$ , the properties of the systems. In other words,  $y$  is assumed to have a distribution with unknown parameters  $\theta$ . In this setting, a hypothesis  $H$  is a condition on  $\theta$ . Hypothesis assessment is a way of evaluating the degree to which the observations  $y$  are compatible with  $H$ . The overall process is depicted in Figure 1.
Following our running example, we use the task of answering natural language questions (Clark et al., 2018). While our examples are shown for this particular task, all the ideas are applicable to more general experimental settings.
For this task, the performance metric  $\mathcal{M}(S,x)$  is defined as a binary function indicating whether a system  $S$  answers a given question  $x$  correctly or not. The performance vector  $\mathcal{M}(S,\boldsymbol {x})$  captures the system's accuracy on the entire dataset (cf. Table 1). We assume that each system  $S_{i}$  has an unknown inherent accuracy value, denoted  $\theta_{i}$ . Let  $\pmb {\theta} = [\theta_1,\theta_2]$  denote the unknown inherent accuracy of two systems. In this setup, one might, for instance, be interested in assessing the credibility of the hypothesis  $H$  that  $\theta_{1} < \theta_{2}$ .
Table 2 shows a categorization of statistical tools developed for the assessment of such hypotheses. The two tools on the left are based on frequentist statistics, while the ones on the right are based on Bayesian inference (Kruschke and Liddell, 2018). A complementary categorization of these tools is based on the nature of the results that they provide: the ones on the top encourage binary decision mak
4For simplicity of exposition, we assume the performances of two systems are on a single dataset. However, the discussion also applies to observations on multiple different datasets.
${}^{5}$  Parametric tests assume this distribution,while non- parametric tests do not.
Frequentist	Bayesian
Binary Decision	(Section 2.1)Null-Hypothesis Significance Test	(Section 2.4)Bayes Factor
Uncertainty estimation	(Section 2.2)Confidence Intervals	(Section 2.3)Posterior Intervals
Table 2: Various classes of methods for statistical assessment of hypotheses.
ing, while those on the bottom provide uncertainty around estimates. We discuss all four classes of tests in the following sub-sections.
2.1 Null-Hypothesis Significance Testing
In frequentist hypothesis testing, there is an asymmetric relationship between two hypotheses. The hypothesis formulated to be rejected is usually called the null-hypothesis  $H_0$ . For instance, in our example  $H_0$ :  $\theta_1 = \theta_2$ . A decision procedure is devised by which, depending on  $\pmb{y}$ , the null-hypothesis will either be rejected in favor of  $H_1$ , or the test will stay undecided.
A key notion here is  $p$ -value, the probability, under the null-hypothesis  $H_0$ , of observing an outcome at least equal to or extreme than the empirical observations  $y$ . To apply this notion on a set of observations  $y$ , one has to define a function that maps  $y$  to a numerical value. This function is called the test statistic  $\delta(\cdot)$  and it formalizes the interpretation of extremeness. Concretely,  $p$ -value is defined as,
$$
\mathbb {P} (\delta (Y) \geq \delta (\boldsymbol {y}) | H _ {0}) \tag {1}
$$
In this notation,  $Y$  is a random variable over possible observations and  $\delta(\boldsymbol{y})$  is the empirically observed value of the test statistic.
A large  $p$ -value implies that the data could easily have been observed under the null-hypothesis. Therefore, a lower  $p$ -value is used as evidence towards rejecting the null-hypothesis.
Example 1 (Assessment of  $H1$ ) We
form a null-hypothesis using the accuracy of the two systems (Table 1) using a one-sided  $z$ -test with  $\delta(\pmb{y}) \triangleq (1/n)\sum_{i=1}^{n}\left[\mathcal{M}(S_1, x_i) - \mathcal{M}(S_2, x_i)\right]$ . We formulate a null-hypothesis against the claim of  $S_1$  having strictly better accuracy than  $S_2$ . This results in a p-value of 0.0037 (details in §A.1) and can be interpreted as the following:
if the systems have inherently identical accuracy values, the probability of observing a superiority at least as extreme as our observations is 0.0037. For a significance level of 0.05 (picked before the test) this  $p$ -value is small enough to reject the null-hypothesis.
aThe choice of this test is based on an implicit assumption that two events corresponding to answering two distinct questions, are independent with identical probability, i.e., equal to the inherent accuracy of the system. Hence, the number of correct answers follows a binomial distribution. Since, the total number of questions is large, i.e., 2376 in ARC-easy, this distribution can be approximated with a normal distribution. It is possible to use other tests with less restrictive assumptions (see Dror et al. (2018)), but for the sake of simplicity we use this test to illustrate core ideas of "p-value" analysis.
This family of the tests is thus far the most widely used tool in NLP research. Each variant of this test is based on some assumptions about the distribution of the observations, under the null-hypothesis, and an appropriate definition of the test statistics  $\delta(\cdot)$ . Since a complete exposition of such tests is outside the scope of this work, we encourage interested readers to refer to the existing reviews, such as Dror et al. (2018).
2.2 Confidence Intervals
Confidence Intervals (CIs) are used to express the uncertainty of estimated parameters. In particular, the  $95\%$  CI is the range of values for parameter  $\theta$  such that the corresponding test based on  $p$ -value is not rejected:
$$
\mathbb {P} (\delta (Y) \geq \delta (\boldsymbol {y}) | H _ {0} (\boldsymbol {\theta})) \geq 0. 0 5. \tag {2}
$$
In other words, the confidence interval merely asks which values of the parameter  $\theta$  could be used, before the test is rejected.
Example 2 (Assessment of H2) Consider the same setting as in Example 1. According to Table 1, the estimated value of the accuracy differences (maximum-likelihood estimates) is  $\theta_{1} - \theta_{2} = 0.035$ . A  $95\%$  CI of this quantity provides a range of values that are not rejected under the corresponding null-hypothesis. In particular, a  $95\%$  CI gives  $\theta_{1} - \theta_{2} \in [0.0136, 0.057]$  (details in §A.2). The blue bar in Figure 2 (right) shows the corresponding CI. Notice that the conclusion of Example 1 is compatible with this CI; the null-hypothesis  $\theta_{1} = \theta_{2}$  which got rejected is not included in the CI.
5718
2.3 Posterior Intervals
Bayesian methods focus on prior and posterior distributions of  $\theta$ . Recall that in a typical NLP experiment, these parameters can be, e.g., the actual mean or standard deviation for the performance of a system, as its inherent and unobserved property.
In Bayesian inference frameworks, a priori assumptions and beliefs are encoded in the form of a prior distribution  $\mathbb{P}(\pmb{\theta})$  on parameters of the model. In other words, a prior distribution describes the common belief about the parameters of the model. It also implies a distribution over possible observations. For assessing hypotheses  $H3$  and  $H4$  in our running example, we will simply use the uniform prior, i.e., the inherent accuracy is uniformly distributed over  $[0,1]$ . This corresponds to having no prior belief about how high or low the inherent accuracy of a typical QA system may be.
In general, the choice of this prior can be viewed as a compromise between the beliefs of the analyzer and those of the audience. The above uniform prior, which is equivalent to the Beta(1,1) distribution, is completely non-committal and thus best suited for a broad audience who has no reason to believe an inherent accuracy of 0.8 is more likely than 0.3. For a moderately informed audience that already believes the inherent accuracy is likely to be widely distributed but centered around 0.67, the analyzer may use a Beta(3,1.5) prior to evaluate a hypothesis. Similarly, for an audience that already believes the inherent accuracy to be highly peaked around 0.75, the analyzer may want to use a Beta(9,3) prior. Formally, one incorporates  $\theta$  in a hierarchical model in the form of a likelihood function  $\mathbb{P}(\pmb {y}|\pmb {\theta})$ . This explicitly models the underlying process that connects the latent parameters to the observations. Consequently, a posterior distribution is inferred using the Bayes rule and conditioned on the observations:  $\mathbb{P}(\pmb {\theta}|\pmb {y}) = \frac{\mathbb{P}(\pmb{y}|\pmb{\theta})\mathbb{P}(\pmb{\theta})}{\mathbb{P}(\pmb{y})}$ .
The posterior distribution is a combined summary of the data and prior information, about likely values of  $\theta$ . The mode of the posterior (maximum a posteriori) can be seen as an estimate for  $\theta$ . Additionally, the posterior can be used to describe the uncertainty around the mode.
While the posterior distribution can be analytically calculated for simple models, it is not so straightforward for general models. Fortunately,
recent advances in hardware, Markov Chain Monte Carlo (MCMC) techniques (Metropolis et al., 1953; Gamerman and Lopes, 2006), and probabilistic programming<sup>7</sup> allow sufficiently-accurate numerical approximations of posteriors.
One way to summarize the uncertainty around the point estimate of parameters is by marking the span of values that cover  $\alpha \%$  of the most-credible density in the posterior distribution (e.g.,  $\alpha = 95\%$ ). This is called Highest Density Intervals (HDIs) or Bayesian Confidence Intervals (Oliphant, 2006) (not to be confused with CI, in §2.2).
Recall that a hypothesis  $H$  is a condition on  $\pmb{\theta}$  (see Figure 1). Therefore, given the posterior  $\mathbb{P}(\pmb {\theta}|\pmb {y})$ , one can calculate the probability of  $H$ , as a probabilistic event, conditioned on  $\pmb{y}$ :  $\mathbb{P}(H|\pmb {y})$ .
For example in an unpaired  $t$ -test,  $H_0$  is the event that the means of two groups are equal. Bayesian statisticians usually relax this strict equality  $\theta_{1} = \theta_{2}$  and instead evaluate the credibility of  $|\theta_{1} - \theta_{2}| < \varepsilon$  for some small value of  $\varepsilon$ . The intuition is that when  $\theta_{1}$  and  $\theta_{2}$  are close enough they are practically equivalent. This motivates the definition of Region Of Practical Equivalence (ROPE): An interval around zero with "negligible" radius. The boundaries of ROPE depend on the application, the meaning of the parameters and its audience. In our running example, a radius of one percent for ROPE implies that improvements less than 1 percent are not considered notable. For a discussion on setting ROPE see Kruschke (2018).
These concepts give researchers the flexibility to define and assess a wide range of hypotheses. For instance, we can address  $H3$  (from Introduction) and its different variations that can be of interest depending on the application. The analysis of  $H3$  is depicted in Figure 2 and explained next. $^8$
Example 3 (Assessment of H3) Recall the setting from previous examples. The left panel of Figure 2 shows the prior on the latent accuracy of the systems and their differences (further details on the hierarchical model in §A.3.) We then obtain the posterior distribution (Figure 2, right), in this case via numerical methods.
Notice that one can read the following conclusion: with probability 0.996, the hypothe
<sup>6</sup>We use  $\mathbb{P}(x)$  in its most general form, to denote the Probability Mass Function for discrete variables and the Probability Density Function for continuous variables.
${}^{7}$  Pymc3 (in Python) and JAGS & STAN (in R) are among the commonly-used packages for this purpose.
<sup>8</sup>Figure 2 can be readily reproduced via the accompanying software, HyBayes.
5719
Figure 2: Left: Prior distributions of two systems (bottom row) and their difference (top row). Right: Posterior distributions of two systems (bottom row) and their difference (top row) after observing the performances on ARC-easy dataset. Note the posterior HDI estimate, (0.00939, 0.0612). Here we assume at least one percent accuracy difference to be considered practically different. Hence, we indicate the interval  $(-0.01, 0.01)$  as ROPE ( $\S 2.3$ ).