SlowGuess/NAACL_2022_OCR huggingface dataset:
Ran Zmigrod
Tim Vieira
Ryan Cotterell
rz279@cam.ac.uk tim.f.vieira@gmail.com
ryan.cotterell@inf.ethz.ch
Significance testing—especially the paired-permutation test—has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in  $\mathcal{O}(GN(\log GN)(\log N))$  time where  $N$  is the dataset size and  $G$  is the range of the test statistic. We found that our exact algorithm was  $10x$  faster than the Monte Carlo approximation with 20000 samples on a common dataset.
https://github.com/rycolab/paired-perm-test
How confident can we be that System  $U$  is more accurate than System  $V$ ? Questions of this form are widespread in natural language processing (Dietterich, 1998; Koehn, 2004; Ojala and Garriga, 2010; Clark et al., 2011; Berg-Kirkpatrick et al., 2012; Dror et al., 2018) and statistical hypothesis testing provides answers (Lehmann and Romano, 2005). In this paper, we study the paired-permutation test (Good, 2000)—a commonly used hypothesis test in NLP because it makes no assumptions on the distribution of the data or the evaluation metric used to compare the two systems (Yeh, 2000; Dror et al., 2018, 2020; Deutsch et al., 2021). The paired-permutation test checks whether a test statistic is significant by evaluating the probability that a value at least as large as the observed statistic would occur if system outputs were randomly swapped. Thus, an exact algorithm for evaluating the paired-permutation test involves a summation over all  $2^{N}$  possible swaps. Without
Figure 1: Runtime comparison of exactperm_test using convolve_DP and convolve_FFT, and monte_carlo as a function of the number of entries in the dataset. See for  $\S 5$  for experimental details.
any assumptions on the test statistic, we can only exactly compute this sum in  $\mathcal{O}(2^N)$  time. Thus, practitioners often resort to running a Monte Carlo (MC) approximation which replaces the summation with  $K\ll 2^{N}$  randomly sampled swaps. Although the MC approximation is often practical, it unfortunately, introduces additional error when determining the significance of a test (Serlin, 2000; Koehler et al., 2009).
This paper proposes a family of additively structured, integer-valued test statistics. Test statistics of this form admit an efficient exact algorithm that leverages the fast Fourier transform (Cooley and Tukey, 1965; Cormen et al., 2022) to run in  $\mathcal{O}(GN(\log GN)(\log N))$  time where  $N$  is the size of the dataset and  $G$  is the range of the test statistic. We compare the efficiency of our exact method to the MC approximation for comparing part-of-speech taggers on the Universal Dependency Dataset (Nivre et al., 2018). Surprisingly, our exact algorithm is faster than MC approximation: given 10000 sentences, our algorithm is 10x faster than MC with 20000 samples and 3x faster than MC with 5000 samples, taking  $\approx 0.1$  seconds.
Human Language Technologies, pages 4894 - 4902
2 Paired-Permutation Testing
The paired-permutation test (Good, 2000), the focus of this work, is a common null hypothesis significance test that has a natural application to many problems in NLP (Peyrard et al., 2021). The test attempts to reject the null hypothesis, described below, at significance level  $\alpha$ ; typically  $\alpha = 0.05$ .
Preliminaries. Suppose we want to compare the performances of two systems  $U$  and  $V$  where each system was evaluated on a dataset of the same  $N$  entries. We place the entries of  $U$  and  $V$  into a pair of arrays of length  $N$  denoted  $\mathbf{u}$  and  $\mathbf{v}$ .
The null hypothesis. The goal of a paired-permutation test is to test whether the entries  $\mathbf{u}$  and  $\mathbf{v}$  are independent of the labels  $U$  and  $V$  themselves. The reason that this is the question we ought to care about is that, fundamentally, if the system label (in this case,  $U$  or  $V$ ) provides no information (in the sense of mutual information) about the entry, then we should not prefer one system to another. And, from basic information theory, we know that two random variables (RVs) have no mutual information iff they are independent. So, independence of the system's label and the system's set of entries is the right thing to inquire about. In the language of frequentist testing, the hypothesis that a system's labels and individual entries are independent is known as the null hypothesis. And, under a paired-permutation test, the goal is to ascertain whether the data (the observed entries  $\mathbf{u}$  and  $\mathbf{v}$ ) provide enough evidence to reject the null hypothesis, i.e., to conclude that the label of a system shares information with the quality of its individual entries, and are indeed dependent.
The null distribution. Next, in order to attempt to reject the null hypothesis, we require a distribution over (hypothetical) pairs of entries  $\mathbf{u}'$  and  $\mathbf{v}'$  whose individual entries are independent of the system labels  $U$  and  $V$ , which is achieved through the construction of RVs  $\mathbf{U}^{\emptyset}$  and  $\mathbf{V}^{\emptyset}$ , whose joint distribution can be used to sample our hypothetical  $\mathbf{u}'$  and  $\mathbf{v}'$ . Traditionally,  $\mathbb{P}[\mathbf{U}^{\emptyset}, \mathbf{V}^{\emptyset}]$  is referred to as the null distribution. A paired-permutation test provides a simple recipe for constructing such an RV pair. The first step is to make an entry-wise independence assumption: we define the joint probability  $\mathbb{P}[\mathbf{U}^{\emptyset}, \mathbf{V}^{\emptyset}] \stackrel{\mathrm{def}}{=} \prod_{n=1}^{N} \mathbb{P}[\mathbf{U}_n^{\emptyset}, \mathbf{V}_n^{\emptyset}]$ . This means that the prediction a system makes for the  $n^{\text{th}}$  entry is independent of the  $m^{\text{th}}$  entry when
$n \neq m$ . In the second step, we further define the entry-wise joint distribution as
$$
\mathbb {P} \left[ \mathrm {U} _ {n} ^ {\varnothing} = u _ {n}, \mathrm {V} _ {n} ^ {\varnothing} = v _ {n} \right] \stackrel {\text {d e f}} {=} \frac {1}{2} \quad (\text {s t a y}) \tag {1a}
$$
$$
\mathbb {P} \left[ \mathrm {U} _ {n} ^ {\varnothing} = v _ {n}, \mathrm {V} _ {n} ^ {\varnothing} = u _ {n} \right] \stackrel {\text {d e f}} {=} \frac {1}{2} \quad (\text {s w a p}) \tag {1b}
$$
In words,  $\mathbb{P}[\mathbf{U}_n^{\emptyset},\mathbf{V}_n^{\emptyset}]$  is a uniform distribution over swapping  $U$  and  $V$  's prediction for the  $n^{\mathrm{th}}$  entry. All in all, this definition of  $\mathbb{P}[\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset}]$  as the null distribution gives us a uniform distribution over all  $2^{N}$  ways swapping of the labels and the individual entries of the observed predictions u and v. And, importantly, the joint distribution  $\mathbb{P}[\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset}]$ , encodes the fact that the sampled entries are independent of the system label.
The test statistic and the  $p$ -value. The final ingredient we need in a null hypothesis test is a test statistic, whose job it is to provide a summary of samples  $(\mathbf{u}',\mathbf{v}')\sim \mathbb{P}[\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset}]$  and thereby facilitate comparison of samples from the null distribution  $\mathbb{P}[\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset}]$  and the observed entries  $(\mathbf{u},\mathbf{v})$ . In this work, we will define a test statistic as function  $t(\mathbf{u},\mathbf{v})$ . In principle, we can choose any test statistic  $t$  that allows us to distinguish  $\mathbf{u}$  and  $\mathbf{v}$ , i.e., we have  $t(\mathbf{u},\mathbf{v}) = 0\iff \mathbf{u} = \mathbf{v}$ . Now, given observed entries  $\mathbf{u}$  and  $\mathbf{v}$ , the  $p$ -value is defined as
$$
p = \mathbb {P} \left[ t \left(\mathbf {U} ^ {\varnothing}, \mathbf {V} ^ {\varnothing}\right) \geq \bar {\xi} \right] \tag {2}
$$
where  $\overline{\xi} \stackrel{\mathrm{def}}{=} t(\mathbf{u},\mathbf{v})$  is the observed effect. In words, the  $p$ -value is the probability of observing a test statistic  $t(\mathbf{u}',\mathbf{v}')$  with a value as large as  $t(\mathbf{u},\mathbf{v})$  where  $(\mathbf{u}',\mathbf{v}') \sim \mathbb{P}[\mathbf{U}^{\varnothing},\mathbf{V}^{\varnothing}]$  are sampled from the null distribution. Recall that the system labels and entries are independent under the null distribution by construction, so the  $p$ -value tells us, under the independence assumption, how likely such a large test statistic would have been observed. The test says that we have sufficient evidence to reject the null hypothesis when  $p < \alpha$ . These concepts are depicted in Fig. 2.
Figure 2: Depiction of the pmf of the test statistic  $t$ , the  $p$ -value, and the observed value  $\overline{\xi}$ .
3 Structured Test Statistics
We now discuss a common special case of the paired-permutation test where the test statistic has a particular structure. In §4, we show how to exploit this structure to develop an efficient algorithm to exactly compute the test. The specific assumption we make is that the test statistic is an integer-valued additively decomposable function. Formally, this assumption means that we can rewrite  $t$  as follows
$$
t (\mathbf {u}, \mathbf {v}) \stackrel {\text {d e f}} {=} h (\boldsymbol {g} (\mathbf {u}, \mathbf {v})) \tag {3}
$$
for any function  $h$  and additively decomposable function  $\pmb{g}(\mathbf{u},\mathbf{v})\stackrel {\mathrm{def}}{=}\sum_{n = 1}^{N}g(u_n,v_n)$  such that  $g$  is an integer-valued function with a range of size  $\mathcal{O}(G)$ . The structure of (3) will allow us to derive an efficient algorithm for evaluating  $\mathbb{P}[t(\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset})] = \mathbb{P}\Big[h\Big(\sum_{n = 1}^{N}g(u_n,v_n)\Big)\Big]$ . We now dissect this equation. Each summand  $g(u_{n},v_{n})$  can take on one of two values  $\vec{\xi}_n\stackrel {\mathrm{def}}{=}g(u_n,v_n)$  and  $\xi_{n}\stackrel {\mathrm{def}}{=}g(v_{n},u_{n})$  with equal probability. We rewrite the sum  $\sum_{n = 1}^{N}g(u_{n},v_{n})$  as  $\mathrm{S}\stackrel {\mathrm{def}}{=}\sum_{n = 1}^{N}\mathrm{Z}_{n}$  where  $\mathbf{Z}_n$  are uniform RVs over the set  $\{\overrightarrow{\xi_n},\overrightarrow{\xi_n}\}$ . Each  $\mathbf{Z}_n$  has probability mass function (PMF)
$$
\begin{array}{l} f _ {n} (z) \stackrel {\text {d e f}} {=} \operatorname {p m f} _ {\mathrm {Z}} (\overrightarrow {\xi_ {n}}, \overleftarrow {\xi_ {n}}) (z) (4a) \\ \stackrel {\text {d e f}} {=} \left\{ \begin{array}{l l} \frac {1}{2} \mathbb {1} \left[ z \in \left\{\overrightarrow {\xi_ {n}}, \overleftarrow {\xi_ {n}} \right\} \right] & \mathbf {i f} \overrightarrow {\xi_ {n}} \neq \overleftarrow {\xi_ {n}} \\ \mathbb {1} \left[ z = \overrightarrow {\xi_ {n}} \right] & \text {o t h e r w i s e} \end{array} \right. (4b) \\ \end{array}
$$
The domain of each PMF,  $\mathrm{dom}(f_n)$ , contains at most two elements. Let  $S \stackrel{\mathrm{def}}{=} \mathrm{dom}(\mathrm{S})$ . Clearly,  $|\mathcal{S}| = \mathcal{O}(GN)$  as we have a sum over  $N$  RVs  $Z_{n}$  each with domain size  $\mathcal{O}(G)$ . The following theorem shows that we can evaluate  $\mathbb{P}[t(\mathbf{U}^{\emptyset}, \mathbf{V}^{\emptyset})]$  from the distribution of S, which we will show in the next section is efficient to compute.
Theorem 1. For any test statistic  $t$  that factorizes as in (3) with  $h$  and  $g$ , the distribution of the test statistic under the null distribution decomposes as
$$
\mathbb {P} \left[ t \left(\mathbf {U} ^ {\varnothing}, \mathbf {V} ^ {\varnothing}\right) \right] = \mathbb {P} [ h (\mathrm {S}) ] \tag {5}
$$
$$
\begin{array}{l} \mathbb {P} \left[ t \left(\mathbf {U} ^ {\varnothing}, \mathbf {V} ^ {\varnothing}\right) \right] = \mathbb {P} \left[ h \left(\sum_ {n = 1} ^ {N} g \left(\mathrm {U} _ {n} ^ {\varnothing}, \mathrm {V} _ {n} ^ {\varnothing}\right)\right) \right] (6a) \\ = \mathbb {P} \left[ h \left(\sum_ {n = 1} ^ {N} Z _ {n}\right) \right] (6b) \\ = \mathbb {P} [ h (\mathrm {S}) ] (6c) \\ \end{array}
$$
1: def monte_carlo(u, v, g, h, K):
2: for  $n = 1$  to  $N$ :
3:  $\overrightarrow{\xi_n} \gets g(u_n, v_n) \quad \triangleright$  Local effect (stay)
4:  $\overleftarrow{\xi_n}\gets g(\overrightarrow{v_n,u_n})\qquad \triangleright$  Local effect (swap)
5:  $f_{n}\gets \mathrm{pmf}_{\mathbb{Z}}\left(\overrightarrow{\xi_{n}},\overleftarrow{\xi_{n}}\right)$
6:  $\overline{\xi} \gets h\left(\sum_{n=1}^{N} \overrightarrow{\xi_n}\right) \quad \triangleright$  Compute observed effect
7: ▷ Sample K random stay or swap actions from each local PMf  $f_{n}$ .
8:  $z_{n}^{(k)}\sim f_{n}$  for  $n = 1$  to  $N,k = 1$  to  $K$
9: return  $\frac{1}{K}\sum_{k = 1}^{K}\mathbb{1}\left[h\left(\sum_{n = 1}^{N}z_n^{(k)}\right)\geq \overline{\xi}\right]$
Algorithm 1: Monte Carlo approximation algorithm for the paired-permutation test.
Example. A common test statistic is the difference in accuracy, in which each entry  $u_{n} \in \{1, \dots, C\}^{G}$  where  $C$  is the number of classes and in this case,  $G$  is the maximum length of an entry sequence (or one if each entry has a binary accuracy value). Then  $g(u_{n}, v_{n}) \in \{-G, \dots, G\}$  is the difference in the number of correct predictions between individual entries  $u_{n}$  and  $v_{n}$ . We can additionally define the function  $h$  as either  $h(x) = x$  or  $h(x) = |x|$  depending on whether we want a one-tailed or two-tailed significance test.
A Monte Carlo paired-permutation test. To the best of our knowledge, no practical exact algorithm for the paired-permutation test has been given in the literature. Thus, most practical implementations of the paired-permutation test use an MC approximation, whereby one randomly samples from S to approximate  $\mathbb{P}[\mathbf{U}^{\emptyset},\mathbf{V}^{\emptyset}]$ . We give this MC algorithm as monte_carlo in Alg 1 which runs in  $\mathcal{O}(KN)$  time where  $K$  is the number of samples taken.
4 An Exact Paired-Permutation Test
In this section, we describe two exact, efficient algorithms for computing the  $p$ -value under the paired-permutation test for any structured test statistic (see (3)).<sup>1</sup> Our algorithms hinge on an important theorem in probability: The PMF of the sum of independent events is the convolution of their individual PMFs (Ross, 2008, p. 252). Let  $f_{\mathrm{S}}$  denote the PMF
<sup>1</sup>In App. B, we extend our algorithms to work with test statistics that use  $m$  additively decomposable scoring functions, e.g., difference in  $F_{1}$  scores.
1: def exactperm_test(u,v,g,h):
2: for  $n = 1$  to  $N$ :
3:  $\overrightarrow{\xi_n} \gets g(u_n, v_n) \quad \triangleright$  Local effect (stay)
4:  $\overleftarrow{\xi_n}\gets g(\overrightarrow{v_n,u_n})$  ▷ Local effect (swap)
5:  $f_{n}\gets \mathrm{pmf}_{\mathrm{Z}}\left(\overrightarrow{\xi_{n}},\overleftarrow{\xi_{n}}\right)$
6:  $\overline{\xi} \gets h\left(\sum_{n=1}^{N} \overrightarrow{\xi_n}\right) \quad \triangleright$  Compute observed effect
7:  $f_{\mathrm{S}}\gets f_1\star \dots \star f_N$  ▷ Convolve the  $f_{n}$  's
8:  $\triangleright$  Sum-up the PMF to get  $p$
9: return  $\sum_{\xi \in \mathcal{S}}f_{\mathrm{S}}(\xi)$  1  $\mathbb{1}[h(\xi)\geq \overline{\xi} ]$
Algorithm 2: Compute the exact  $p$  value for the paired-permutation test for structured test statistics.
of S. Since RVs  $Z_{n}$  are independent, we have that
$$
\begin{array}{l} \mathbb {P} [ h (\mathrm {S}) \geq \bar {\xi} ] (7a) \\ = \sum_ {\xi \in \mathcal {S}} f _ {\mathrm {S}} (\xi) \mathbb {1} [ h (\xi)) \geq \bar {\xi} ] (7b) \\ = \sum_ {\xi \in \mathcal {S}} \left(f _ {1} \star \dots \star f _ {N}\right) (\xi) \mathbb {1} [ h (\xi)) \geq \bar {\xi} ] \quad (7 c) \\ \end{array}
$$
where  $\star$  is the discrete convolution operator. For functions  $f_{i}, f_{j} \in S \to \mathbb{R}$ ,  $f_{i} \star f_{j} \in S \to \mathbb{R}$  is given by the following expression
$$
\left(f _ {i} \star f _ {j}\right) (\xi) \stackrel {\text {d e f}} {=} \sum_ {\xi^ {\prime} \in \mathcal {S}} f _ {i} \left(\xi^ {\prime}\right) f _ {j} \left(\xi - \xi^ {\prime}\right) \tag {8}
$$
Pseudocode for this algorithm is given as exact_perm_test in Alg 2. We omit the details of evaluating the convolution in exact_perm_test and discuss methods for efficient convolution in the remainder of this section.
Theorem 2. For any two entries,  $\mathbf{u}$  and  $\mathbf{v}$ , and test statistic  $t$  that factorizes as in (3) with  $h$  and  $g$ , exact_perm_test  $(\mathbf{u},\mathbf{v},g,h)$  returns  $p$  in  $\mathcal{O}(GN + r(G,N))$  time,  $\mathcal{O}(N + s(G,N))$  space. We define  $r(G,N)$  and  $s(G,N)$  as the time and space complexities for constructing  $f_{1} \star \dots \star f_{N}$ .
Proof. The correctness of exact_perm_test is by Theorem 1 and (7c). All lines except for Line 7 and Line 9 require at most  $\mathcal{O}(N)$  time and space. Line 9 runs in  $\mathcal{O}(GN)$  time and  $\mathcal{O}(1)$  space. Thus, exact_perm_test runs in  $\mathcal{O}(N + GN + r(G,N))$  time and  $\mathcal{O}(N + s(G,N))$  space.
The computational question is then: What is the most efficient algorithm for evaluating  $(f_{1} \star \dots \star f_{N})(\xi)$ ? In the following two
1: def convolve_DP  $(f_{1},\dots,f_{N})$  ..
2:  $\mathbf{F} \gets \mathbf{0}$
3:  $\mathrm{F}_0(0)\gets 1$ $\triangleright$  Init:  $\mathbb{P}[0] = 1$  if  $N = 0$
4: for  $n = 1$  to  $N$ :
5:  $\triangleright$  Compute  $\mathrm{F}_n = f_n\star \mathrm{F}_{n - 1}$
6: for  $\xi \in \mathrm{dom}(f_n)$  ..  $\triangleright |\mathrm{dom}(f_n)|\leq 2$
7: for  $\xi' \in \mathrm{dom}(\mathrm{F}_{n-1})$ :
8:  $\mathrm{F}_n(\xi +\xi ') + = f_n(\xi)\cdot \mathrm{F}_{n - 1}(\xi ')$
9: return  $\mathbf{F}_N$
Algorithm 3: Dynamic programming algorithm to compute the pmf of S as the  $N$ -fold convolution of the pmfs  $f_{1}, \ldots, f_{N}$ . Note that we only need to store  $\mathrm{F}_n$  and  $\mathrm{F}_{n-1}$  at any given time.
subsections, we present  $\mathcal{O}(GN^2)$  time and  $\mathcal{O}(GN(\log GN)(\log N))$  time algorithms for performing this  $N$ -fold convolution.
4.1 Convolution by Dynamic Programming
Our first approach builds a dynamic program (DP) that takes advantage of the sparsity of our RVs to efficiently construct the PMF  $f_{\mathrm{S}}$ . We do this by constructing a PMF array  $\mathrm{F}_n \in \mathbb{R}^S$  for  $n \in \{0, \dots, N\}$  (we use  $n = 0$  as an initialisation base case) such that  $\mathrm{F}_n(\xi) = (f_1 \star \dots \star f_n)(\xi)$ . As we apply each convolution, we know that  $f_n$  is only non-zero at  $\vec{\xi}_n$  and  $\vec{\xi}_n$ , and so we can run each convolution in  $\mathcal{O}(GN)$  time. The pseudocode for this approach is given as convolve_DP in Alg 3.
Theorem 3. For any RVs  $Z_{1},\ldots ,Z_{N}$  with PMFs  $f_{1},\ldots ,f_{N}$ , convolve  $\mathsf{DP}(f_n,\dots,f_N)$  returns  $f_{1}\star \dots \star f_{n}$  in  $\mathcal{O}(GN^2)$  time,  $\mathcal{O}(GN)$  space.
Proof. The proof of correctness of convolve_DP is given in App. A. Each  $\mathrm{F}_n$  has  $\mathcal{O}(GN)$  elements and so convolve_DP clearly runs in  $\mathcal{O}(GN^2)$  time. Furthermore, at any iteration, we only require  $\mathrm{F}_n$  and  $\mathrm{F}_{n-1}$  and so we can execute convolve_DP in  $\mathcal{O}(GN)$  space.
4.2 Convolution by FFT
Our second approach uses the fast Fourier transform (FFT; Cooley and Tukey, 1965; Cormen et al., 2022) to evaluate the convolutions. Using this method means that each convolution takes  $\mathcal{O}(GN\log GN)$  time  $\mathcal{O}(GN)$  space. We further exploit the commutativity of convolution to perform the  $N$ -fold convolution in  $\log N$  convolutions using a recursive program. The pseudocode for this approach is given as convolve_FFT in Alg 4.
1: def convolve_FFT  $(f_{1},\dots,f_{N})$  :
2: if  $N = 1$  : return  $f_{1}$
3: return convolve_FFT(f1,...,fN/2)
$$
\star \text {c o n v o l v e} _ {-} \mathsf {F F T} \left(f _ {N / / 2 + 1}, \dots , f _ {N}\right)
$$
Algorithm 4: Recursive algorithm to compute the pmf of S as the  $N$ -fold convolution of the pmfs  $f_{1}, \ldots, f_{N}$ . Here the convolution operation  $(\star)$  runs in  $\mathcal{O}(GN \log GN)$  time thanks to the FFT (Cooley and Tukey, 1965).
Theorem 4. For any RVs  $Z_{1},\ldots ,Z_{N}$  with PMFs  $f_{1},\dots,f_{N}$ , convolve_FFT  $(f_{1},\dots,f_{N})$  returns  $f_{1}\star \dots \star f_{n}$  in  $\mathcal{O}(GN(\log GN)(\log N))$  time,  $\mathcal{O}(GN\log N)$  space.2
Proof. The correctness of convolve_FFT is due to Cooley and Tukey (1965). The recursion of convolve_FFT can be given as
$$
T (N) = 2 T \left(\frac {N}{2}\right) + \mathcal {O} (G N \log G N) \tag {9}
$$
Solving this recursion, we call  $T\mathcal{O}(\log N)$  times. Therefore, the time complexity of convolve_FFT is  $\mathcal{O}(GN(\log GN)(\log N))$ . Similarly, each call requires  $\mathcal{O}(GN)$  space and convolve_FFT has a space complexity of  $\mathcal{O}(GN\log N)$ .
Corollary 1. For any two entries,  $\mathbf{u}$  and  $\mathbf{v}$ , and test statistic  $t$  that factorizes as in (3) with  $h$  and  $g$ , exactperm_test  $(\mathbf{u},\mathbf{v},g,h)$  returns  $p$  in  $\mathcal{O}(GN(\log GN)(\log N))$  time,  $\mathcal{O}(GN)$  space.
Proof. The correctness and complexity bounds are due to Theorem 2 and Theorem 4. Specifically, Line 7 can be executed using convolve_FFT.
We demonstrate the efficiency of our exact algorithms by simulating paired-permutation tests between the accuracy of two systems. In order to have some control over the  $p$ -value,  $N$ , and  $G$  (maximum length of a sentence), we randomly generate our two system outputs from a measured distribution. Specifically, we will use the Stanza $^3$  (Qi et al., 2020) part-of-speech tag accuracy statistics when evaluating on the English Universal Dependencies (UD) test set (Nivre et al., 2018). We sample our outputs from the normal distribution where the mean and standard deviation match the rates
2We note that the log  $N$  factor in the space complexity may be eliminated by tail recursion elimination (Muchnick, 1998).
<sup>3</sup>The code and pre-trained model are both freely accessible at https://github.com/stanfordnlp/stanza.
Metric	Mean	Standard Dev.
Accuracy	0.9543	0.1116
Sentence length	12.08	10.60
Table 1: Distributions for of accuracy and sentence length for POS tagging using Stanza (Qi et al., 2020) on the English UD test dataset (Nivre et al., 2018).
of Stanza's observed accuracy. We further sample the length of each sample sentence according to the distribution of lengths in the test set. These distributions are provided in Tab. 1.
We show that, empirically, the exact test is more efficient than the MC approximation; this is evinced in Fig. 1 where we have compared the runtime of exactperm_test using convolve_DP and convolve_FFT against monte_carlo for various sample sizes  $(K\in$ $\{5000,10000,20000,40000\})$  .4 We note that using convolve_DP is already more efficient than running monte_carlo with  $K = 40000$  and  $K = 20000$  (up to  $N\approx 8000$  ).5 Furthermore, convolve_FFT is much faster and we observe a speed-up between  $3\mathrm{x}$  and  $30\mathrm{x}$  ,depending on the number of samples  $K$  .Indeed, using convolve_FFT allows us to perform an exact paired-permutation test for  $N = 10000$  in approximately one-tenth of a second.
We presented an algorithm to compute the exact  $p$ -value of a paired-permutation test for the case of a family of structured test statistics, including the difference in accuracy. Our algorithm runs in  $\mathcal{O}(GN(\log GN)(\log N))$  time and requires  $\mathcal{O}(GN)$  space. We empirically show that our exact algorithm is faster than Monte Carlo approximation techniques. The theory of our work is extensible to a more general class of test statistics which we discuss in App. B. We hope that this work encourages the use of exact paired-permutation tests in future NLP research.
We foresee no ethical concerns in this work.