Venue: NAACL 2022

# Exact Paired-Permutation Testing for Structured Test Statistics

Ran Zmigrod\({}^{\circledS}\) Tim Vieira\({}^{\circledS}\) Ryan Cotterell\({}^{\circledS}\)

University of Cambridge \({}^{\circledS}\) Johns Hopkins University \({}^{\circledS}\) ETH Zurich rz279@cam.ac.uk tim.f.vieira@gmail.comryan.cotterell@inf.ethz.ch

###### Abstract

Significance testing--especially the paired-permutation test--has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in \(\mathcal{O}(GN(\log GN)(\log N))\) time where \(N\) is the dataset size and \(G\) is the range of the test statistic. We found that our exact algorithm was \(10\)x faster than the Monte Carlo approximation with \(20000\) samples on a common dataset.

* [https://github.com/rycolab/paired-perm-test](https://github.com/rycolab/paired-perm-test)

## 1 Introduction

How confident can we be that System \(U\) is more accurate than System _V_? Questions of this form are widespread in natural language processing Dietterich (1998); Koehn (2004); Ojala and Garriga (2010); Clark et al. (2011); Berg-Kirkpatrick et al. (2012); Dror et al. (2018) and statistical hypothesis testing provides answers Lehmann and Romano (2005). In this paper, we study the paired-permutation test Good (2000)--a commonly used hypothesis test in NLP because it makes no assumptions on the distribution of the data or the evaluation metric used to compare the two systems Yeh (2000); Dror et al. (2018, 2020); Deutsch et al. (2021). The paired-permutation test checks whether a test statistic is significant by evaluating the probability that a value at least as large as the observed statistic would occur if system outputs were randomly swapped. Thus, an exact algorithm for evaluating the paired-permutation test involves a summation over all \(2^{N}\) possible swaps. Without any assumptions on the test statistic, we can only exactly compute this sum in \(\mathcal{O}(2^{N})\) time. Thus, practitioners often resort to running a Monte Carlo (MC) approximation which replaces the summation with \(K\ll 2^{N}\) randomly sampled swaps. Although the MC approximation is often practical, it unfortunately,introduces additional error when determining the significance of a test Serlin (2000); Koehler et al. (2009).

This paper proposes a family of additively structured, integer-valued test statistics. Test statistics of this form admit an efficient exact algorithm that leverages the fast Fourier transform Cooley and Tukey (1965); Cormen et al. (2022) to run in \(\mathcal{O}(GN(\log GN)(\log N))\) time where \(N\) is the size of the dataset and \(G\) is the range of the test statistic. We compare the efficiency of our exact method to the MC approximation for comparing part-of-speech taggers on the Universal Dependency Dataset Nivre et al. (2018). Surprisingly, our _exact_ algorithm is faster than MC approximation: given \(10000\) sentences, our algorithm is \(10\)x faster than MC with \(20000\) samples and 3x faster than MC with \(5000\) samples, taking \(\approx\) 0.1 seconds.

Figure 1: Runtime comparison of exact_perm_test using convolve_DP and convolve_FFT, and monte_carlo as a function of the number of entries in the dataset. See for §5 for experimental details.





## 2 Paired-Permutation Testing

The **paired-permutation test**[14], the focus of this work, is a common null hypothesis significance test that has a natural application to many problems in NLP [30]. The test attempts to reject the null hypothesis, described below, at significance level \(\alpha\); typically \(\alpha=0.05\).

Preliminaries.Suppose we want to compare the performances of two systems \(U\) and \(V\) where each system was evaluated on a dataset of the same \(N\) entries. We place the entries of \(U\) and \(V\) into a pair of arrays of length \(N\) denoted \({\bf u}\) and \({\bf v}\).

The null hypothesis.The goal of a paired-permutation test is to test whether the entries \({\bf u}\) and \({\bf v}\) are _independent_ of the labels \(U\) and \(V\) themselves. The reason that this is the question we ought to care about is that, fundamentally, if the system label (in this case, \(U\) or _V_) provides no information (in the sense of mutual information) about the entry, then we should _not_ prefer one system to another. And, from basic information theory, we know that two random variables (RVs) have no mutual information iff they are independent. So, independence of the system's label and the system's set of entries is the right thing to inquire about. In the language of frequentist testing, the hypothesis that a system's labels and individual entries are independent is known as the **null hypothesis**. And, under a paired-permutation test, the goal is to ascertain whether the data (the observed entries \({\bf u}\) and \({\bf v}\)) provide enough evidence to _reject_ the null hypothesis, i.e., to conclude that the label of a system shares information with the quality of its individual entries, and are indeed dependent.

The null distribution.Next, in order to attempt to reject the null hypothesis, we require a distribution over (hypothetical) pairs of entries \({\bf u^{\prime}}\) and \({\bf v^{\prime}}\) whose individual entries are independent of the system labels \(U\) and \(V\), which is achieved through the construction of RVs \({\bf U}^{\varnothing}\) and \({\bf V}^{\varnothing}\), whose joint distribution can be used to sample our hypothetical \({\bf u^{\prime}}\) and \({\bf v^{\prime}}\). Traditionally, \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\) is referred to as the **null distribution**. A paired-permutation test provides a simple recipe for constructing such an RV pair. The first step is to make an entry-wise independence assumption: we define the joint probability \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\stackrel{{ \text{def}}}{{=}}\prod_{n=1}^{N}\mathbb{P}[{\rm U}^{\varnothing}_{ n},{\rm V}^{\varnothing}_{n}]\). This means that the prediction a system makes for the \(n^{\text{th}}\) entry is independent of the \(m^{\text{th}}\) entry when \(n\neq m\). In the second step, we further define the entry-wise joint distribution as

\[\mathbb{P}\big{[}{\rm U}^{\varnothing}_{n}=u_{n},{\rm V}^{ \varnothing}_{n}=v_{n}\big{]}\stackrel{{\text{def}}}{{=}}\frac{1} {2}\quad\mbox{$\mathrm{(stay)}$} \tag{1a}\] \[\mathbb{P}\big{[}{\rm U}^{\varnothing}_{n}=v_{n},{\rm V}^{ \varnothing}_{n}=u_{n}\big{]}\stackrel{{\text{def}}}{{=}}\frac{1} {2}\quad\mbox{$\mathrm{(swap)}$} \tag{1b}\]

In words, \(\mathbb{P}[{\rm U}^{\varnothing}_{n},{\rm V}^{\varnothing}_{n}]\) is a uniform distribution over swapping \(U\) and _V_'s prediction for the \(n^{\text{th}}\) entry. All in all, this definition of \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\) as the null distribution gives us a uniform distribution over all \(2^{N}\) ways swapping of the labels and the individual entries of the observed predictions \({\bf u}\) and \({\bf v}\). And, importantly, the joint distribution \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\), encodes the fact that the sampled entries are independent of the system label.

The test statistic and the \(p\)-value.The final ingredient we need in a null hypothesis test is a **test statistic**, whose job it is to provide a summary of samples \(({\bf u^{\prime}},{\bf v^{\prime}})\sim\mathbb{P}[{\bf U}^{\varnothing},{\bf V }^{\varnothing}]\) and thereby facilitate comparison of samples from the null distribution \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\) and the observed entries \(({\bf u},{\bf v})\). In this work, we will define a test statistic as function \(t({\bf u},{\bf v})\). In principle, we can choose _any_ test statistic \(t\) that allows us to distinguish \({\bf u}\) and \({\bf v}\), i.e., we have have \(t({\bf u},{\bf v})=0\iff{\bf u}={\bf v}\). Now, given observed entries \({\bf u}\) and \({\bf v}\), the \(p\)-value is defined as

\[p=\mathbb{P}\big{[}t({\bf U}^{\varnothing},{\bf V}^{\varnothing})\geq\overline {\xi}\big{]} \tag{2}\]

where \(\overline{\xi}\stackrel{{\text{def}}}{{=}}t({\bf u},{\bf v})\) is the **observed effect**. In words, the \(p\)-value is the probability of observing a test statistic \(t({\bf u^{\prime}},{\bf v^{\prime}})\) with a value as large as \(t({\bf u},{\bf v})\) where \(({\bf u^{\prime}},{\bf v^{\prime}})\sim\mathbb{P}[{\bf U}^{\varnothing},{\bf V }^{\varnothing}]\) are sampled from the null distribution. Recall that the system labels and entries are independent under the null distribution by construction, so the \(p\)-value tells us, under the independence assumption, how likely such a large test statistic would have been observed. The test says that we have sufficient evidence to reject the null hypothesis when \(p<\alpha\). These concepts are depicted in Fig. 2.

Figure 2: Depiction of the pmf of the test statistic \(t\), the \(p\)-value, and the observed value \(\overline{\xi}\).





## 3 Structured Test Statistics

We now discuss a common special case of the paired-permutation test where the test statistic has a particular structure. In SS4, we show how to exploit this structure to develop an efficient algorithm to exactly compute the test. The specific assumption we make is that the test statistic is an integer-valued **additively decomposable function**. Formally, this assumption means that we can rewrite \(t\) as follows

\[t({\bf u},{\bf v})\stackrel{{\mbox{\tiny def}}}{{=}}h({\boldsymbol {g}}({\bf u},{\bf v})) \tag{3}\]

for any function \(h\) and additively decomposable function \({\boldsymbol{g}}({\bf u},{\bf v})\stackrel{{\mbox{\tiny def}}}{{= }}\sum_{n=1}^{N}g(u_{n},v_{n})\) such that \(g\) is an integer-valued function with a range of size \({\cal O}(G)\). The structure of (3) will allow us to derive an efficient algorithm for evaluating \(\mathbb{P}[t({\bf U}^{\varnothing},{\bf V}^{\varnothing})]=\mathbb{P}\Big{[} h\Big{(}\sum_{n=1}^{N}g(u_{n},v_{n})\Big{)}\Big{]}\). We now dissect this equation. Each summand \(g(u_{n},v_{n})\) can take on one of two values \(\overleftarrow{\xi_{n}}\stackrel{{\mbox{\tiny def}}}{{=}}g(u_{ n},v_{n})\) and \(\overleftarrow{\xi_{n}}\stackrel{{\mbox{\tiny def}}}{{=}}g(v_{n},u_ {n})\) with equal probability. We rewrite the sum \(\sum_{n=1}^{N}g(u_{n},v_{n})\) as \(\mathrm{S}\!\stackrel{{\mbox{\tiny def}}}{{=}}\!\sum_{n=1}^{N} \mathrm{Z}_{n}\) where \(\mathrm{Z}_{n}\) are uniform RVs over the set \(\{\overrightarrow{\xi_{n}},\overleftarrow{\xi_{n}}\}\). Each \(\mathrm{Z}_{n}\) has probability mass function (PMF)

\[f_{n}(z) \stackrel{{\mbox{\tiny def}}}{{=}}\operatorname*{pmf }_{\mathrm{Z}}(\overrightarrow{\xi_{n}},\overleftarrow{\xi_{n}})(z) \tag{4a}\] \[\stackrel{{\mbox{\tiny def}}}{{=}}\begin{cases} \frac{1}{2}\mathbbm{1}\Big{[}z\in\{\overrightarrow{\xi_{n}},\overleftarrow{ \xi_{n}}\}\Big{]}&\text{\bf if $\overrightarrow{\xi_{n}}\neq\overleftarrow{\xi_{n}}$}\\ \mathbbm{1}\Big{[}z=\overrightarrow{\xi_{n}}\Big{]}&\text{otherwise}\end{cases} \tag{4b}\]

The domain of each PMF, \(\operatorname*{dom}(f_{n})\), contains at most two elements. Let \({\mathcal{S}}\stackrel{{\mbox{\tiny def}}}{{=}}\operatorname*{ dom}(\mathrm{S})\). Clearly, \(|{\mathcal{S}}|={\cal O}(GN)\) as we have a sum over \(N\) RVs \(\mathrm{Z}_{n}\) each with domain size \({\cal O}(G)\). The following theorem shows that we can evaluate \(\mathbb{P}[t({\bf U}^{\varnothing},{\bf V}^{\varnothing})]\) from the distribution of \(\mathrm{S}\), which we we will show in the next section is efficient to compute.

**Theorem 1**.: _For any test statistic \(t\) that factorizes as in (3) with \(h\) and \(g\), the distribution of the test statistic under the null distribution decomposes as_

\[\mathbb{P}\big{[}t({\bf U}^{\varnothing},{\bf V}^{\varnothing})\big{]}=\mathbb{ P}[h(\mathrm{S})] \tag{5}\]

Proof.: \[\mathbb{P}\big{[}t({\bf U}^{\varnothing},{\bf V}^{\varnothing}) \big{]} =\mathbb{P}\Bigg{[}h\bigg{(}\sum_{n=1}^{N}g({\rm U}^{\varnothing} _{n},{\rm V}^{\varnothing}_{n})\bigg{)}\Bigg{]}\] (6a) \[=\mathbb{P}\Bigg{[}h\bigg{(}\sum_{n=1}^{N}\mathrm{Z}_{n}\bigg{)} \Bigg{]}\] (6b) \[=\mathbb{P}[h(\mathrm{S})]\] (6c)

Example.A common test statistic is the difference in accuracy, in which each entry \(u_{n}\in\left\{1,...,C\right\}^{G}\) where \(C\) is the number of classes and in this case, \(G\) is the maximum length of an entry sequence (or one if each entry has a binary accuracy value). Then \(g(u_{n},v_{n})\in\left\{-G,...,G\right\}\) is the difference in the number of correct predictions between individual entries \(u_{n}\) and \(v_{n}\). We can additionally define the function \(h\) as either \(h(x)=x\) or \(h(x)=|x|\) depending on whether we want a one-tailed or two-tailed significance test.

A Monte Carlo paired-permutation test.To the best of our knowledge, no practical _exact_ algorithm for the paired-permutation test has been given in the literature. Thus, most practical implementations of the paired-permutation test use an MC approximation, whereby one randomly samples from \(\mathrm{S}\) to approximate \(\mathbb{P}[{\bf U}^{\varnothing},{\bf V}^{\varnothing}]\). We give this MC algorithm as monte_carlo in Alg 1 which runs in \({\cal O}(KN)\) time where \(K\) is the number of samples taken.

## 4 An Exact Paired-Permutation Test

In this section, we describe two exact, efficient algorithms for computing the \(p\)-value under the paired-permutation test for any structured test statistic (see (3)).1 Our algorithms hinge on an important theorem in probability: The PMF of the sum of independent events is the convolution of their individual PMFs (Ross, 2008, p. 252). Let \(f_{\mathrm{S}}\) denote the PMF

of S. Since RVs \({\rm Z}_{n}\) are independent, we have that

\[\mathbb{P}\big{[}h({\rm S})\geq\overline{\xi}\big{]} \tag{7a}\] \[= \sum_{\xi\in\mathcal{S}}f_{\rm S}(\xi)\operatorname{\mathbbm{1}} \!\left[h(\xi)\right)\geq\overline{\xi}\] (7b) \[= \sum_{\xi\in\mathcal{S}}(f_{1}\star\cdots\star f_{N})(\xi) \operatorname{\mathbbm{1}}\!\left[h(\xi)\right)\geq\overline{\xi}\big{]} \tag{7c}\]

where \(\star\) is the discrete **convolution operator**. For functions \(f_{i},f_{j}\in\mathcal{S}\to\mathbb{R}\), \(f_{i}\star f_{j}\in\mathcal{S}\to\mathbb{R}\) is given by the following expression

\[(f_{i}\star f_{j})(\xi)\stackrel{{\text{\tiny def}}}{{=}}\sum_{ \xi^{\prime}\in\mathcal{S}}f_{i}(\xi^{\prime})\,f_{j}(\xi-\xi^{\prime}) \tag{8}\]

Pseudocode for this algorithm is given as exact_perm_test in Alg 2. We omit the details of evaluating the convolution in exact_perm_test and discuss methods for efficient convolution in the remainder of this section.

```
1:defexact_perm_test(u,v,g,h):
2:for\(n=1\)to\(N\):
3:\(\overline{\xi_{n}}\gets g(u_{n},v_{n})\)\(\triangleright\)Local effect (stay)
4:\(\overline{\xi_{n}}\leftarrow g(\widetilde{v_{n}},\widetilde{u_{n}})\)\(\triangleright\)Local effect (swap)
5:\(f_{n}\leftarrow\mathrm{pmf}_{Z}\left(\overline{\xi_{n}},\overline{\xi_{n}}\right)\)
6:\(\overline{\xi}\gets h\Bigg{(}\overset{N}{\underset{n=1}{\sum}}\overline{ \xi_{n}}\Bigg{)}\)\(\triangleright\)Compute observed effect
7:\(f_{\rm S}\gets f_{1}\star...\star f_{N}\)\(\triangleright\)Convolve the \(f_{n}\)'s
8:\(\triangleright\)Sum-up the PMF to get p
9:return\(\sum_{\xi\in\mathcal{S}}f_{\rm S}(\xi)\operatorname{\mathbbm{1}}\!\left[h( \xi)\geq\overline{\xi}\,\right]\)
```

Algorithm 2: Compute the exact \(p\) value for the paired-permutation test for structured test statistics.

**Theorem 2**.: _For any two entries, \({\bf u}\) and \({\bf v}\), and test statistic \(t\) that factorizes as in (3) with \(h\) and \(g\), \(\texttt{exact\_perm\_test}({\bf u},{\bf v},g,h)\) returns \(p\) in \(\mathcal{O}(GN+r(G,N))\) time, \(\mathcal{O}(N+s(G,N))\) space. We define \(r(G,N)\) and \(s(G,N)\) as the time and space complexities for constructing \(f_{1}\star\cdots\star f_{N}\)._

Proof.: The correctness of exact_perm_test is by Theorem 1 and (7c). All lines except for Line 7 and Line 9 require at most \(\mathcal{O}(N)\) time and space. Line 9 runs in \(\mathcal{O}(GN)\) time and \(\mathcal{O}(1)\) space. Thus, exact_perm_test runs in \(\mathcal{O}(N+GN+r(G,N))\) time and \(\mathcal{O}(N+s(G,N))\) space. 

The computational question is then: What is the most efficient algorithm for evaluating \((f_{1}\star\cdots\star f_{N})(\xi)\)? In the following two subsections, we present \(\mathcal{O}(GN^{2})\) time and \(\mathcal{O}(GN(\log GN)(\log N))\) time algorithms for performing this \(N\)-fold convolution.

### Convolution by Dynamic Programming

Our first approach builds a dynamic program (DP) that takes advantage of the sparsity of our RVs to efficiently construct the PMF \(f_{\rm S}\). We do this by constructing a PMF array \({\rm F}_{n}\in\mathbb{R}^{\mathcal{S}}\) for \(n\in\{0,..,N\}\) (we use \(n=0\) as an initialisation base case) such that \({\rm F}_{n}(\xi)=(f_{1}\star\cdots\star f_{n})(\xi)\). As we apply each convolution, we know that \(f_{n}\) is only non-zero at \(\overrightarrow{\xi_{n}}\) and \(\overleftarrow{\xi_{n}}\), and so we can run each convolution in \(\mathcal{O}(GN)\) time. The pseudocode for this approach is given as convolve_DP in Alg 3.

**Theorem 3**.: _For any RVs \({\rm Z}_{1},...,{\rm Z}_{N}\) with PMFs \(f_{1},...,f_{N}\), convolve_DP\((f_{n},...,f_{N})\) returns \(f_{1}\star\cdots\star f_{n}\) in \(\mathcal{O}(GN^{2})\) time, \(\mathcal{O}(GN)\) space._

Proof.: The proof of correctness of convolve_DP is given in App. A. Each \({\rm F}_{n}\) has \(\mathcal{O}(GN)\) elements and so convolve_DP clearly runs in \(\mathcal{O}(GN^{2})\) time. Furthermore, at any iteration, we only require \({\rm F}_{n}\) and \({\rm F}_{n-1}\) and so we can execute convolve_DP in \(\mathcal{O}(GN)\) space. 

### Convolution by FFT

Our second approach uses the fast Fourier transform (FFT; Cooley and Tukey, 1965; Cormen et al., 2022) to evaluate the convolutions. Using this method means that each convolution takes \(\mathcal{O}(GN\log GN)\) time \(\mathcal{O}(GN)\) space. We further exploit the commutativity of convolution to perform the \(N\)-fold convolution in \(\log N\) convolutions using a recursive program. The pseudocode for this approach is given as convolve_FFT in Alg 4 

1: **def\({\sf convolve\_FFT}(f_{1},...,f_{N}):\)**
2. **if**\(N=1:\)**return**\(f_{1}\)
3. **return**\({\sf convolve\_FFT}(f_{1},...,f_{N/2})\)  \(\star\)\({\sf convolve\_FFT}(f_{N/2+1},...,f_{N})\)

Algorithm 4: Recursive algorithm to compute the pmf of S as the \(N\)-fold convolution of the pmfs \(f_{1},...,f_{N}\). Here the convolution operation (\(\star\)) runs in \(\mathcal{O}(GN\log GN)\) time thanks to the FFT [8].

Theorem 5.1: _For any RVs \(\mathrm{Z}_{1},...,\mathrm{Z}_{N}\) with PMFs \(f_{1},...,f_{N}\), \({\sf convolve\_FFT}(f_{1},...,f_{N})\) returns \(f_{1}\star\cdots\star f_{n}\) in \(\mathcal{O}(GN(\log GN)(\log N))\) time, \(\mathcal{O}(GN\log N)\) space.2_

Footnote 2: We note that the \(\log N\) factor in the space complexity may be eliminated by tail recursion elimination [6].

Proof: The correctness of \({\sf convolve\_FFT}\) is due to Cooley and Tukey (1965). The recursion of \({\sf convolve\_FFT}\) can be given as

\[T(N)=2\,T\!\left(\frac{N}{2}\right)+\mathcal{O}(GN\log GN) \tag{9}\]

Solving this recursion, we call \(T\;\mathcal{O}(\log N)\) times. Therefore, the time complexity of \({\sf convolve\_FFT}\) is \(\mathcal{O}(GN(\log GN)(\log N))\). Similarly, each call requires \(\mathcal{O}(GN)\) space and \({\sf convolve\_FFT}\) has a space complexity of \(\mathcal{O}(GN\log N)\). 

Corollary 1: _For any two entries, \({\bf u}\) and \({\bf v}\), and test statistic \(t\) that factorizes as in (3) with \(h\) and \(g\), \({\sf exact\_perm\_test}({\bf u},{\bf v},g,h)\) returns \(p\) in \(\mathcal{O}(GN(\log GN)(\log N))\) time, \(\mathcal{O}(GN)\) space._

Proof: The correctness and complexity bounds are due to Theorem 5.1 and Theorem 5.1. Specifically, Line 7 can be executed using \({\sf convolve\_FFT}\). 

## 5 Experiments

We demonstrate the efficiency of our exact algorithms by simulating paired-permutation tests between the accuracy of two systems. In order to have some control over the \(p\)-value, \(N\), and \(G\) (maximum length of a sentence), we randomly generate our two system outputs from a measured distribution. Specifically, we will use the Stanza3[20] part-of-speech tag accuracy statistics when evaluating on the English Universal Dependencies (UD) test set [14]. We sample our outputs from the normal distribution where the mean and standard deviation match the rates of Stanza's observed accuracy. We further sample the length of each sample sentence according to the distribution of lengths in the test set. These distributions are provided in Tab. 1.

Footnote 3: The code and pre-trained model are both freely accessible at [https://github.com/stanfordnlp/stanza](https://github.com/stanfordnlp/stanza).

We show that, empirically, the exact test is _more efficient_ than the MC approximation; this is evinced in Fig. 1 where we have compared the runtime of \({\sf exact\_perm\_test}\) using \({\sf convolve\_DP}\) and \({\sf convolve\_FFT}\) against \({\sf monte\_carlo}\) for various sample sizes (\(K\in\{5000,10000,20000,40000\}\)).4 We note that using \({\sf convolve\_DP}\) is already more efficient than running \({\sf monte\_carlo}\) with \(K=40000\) and \(K=20000\) (up to \(N\approx 8000\)).5 Furthermore, \({\sf convolve\_FFT}\) is _much_ faster and we observe a speed-up between \(3\)x and \(30\)x, depending on the number of samples \(K\). Indeed, using \({\sf convolve\_FFT}\) allows us to perform an exact paired-permutation test for \(N=10000\) in approximately one-tenth of a second.

Footnote 4: The experiment used an Apple M1 Max processor.

Footnote 5: We note that the average test set size of the \(129\) UD treebanks we examined is just over \(1000\) sentences, and only three treebanks had more than \(6000\) sentences.

## 6 Conclusion

We presented an algorithm to compute the exact \(p\)-value of a paired-permutation test for the case of a family of structured test statistics, including the difference in accuracy. Our algorithm runs in \(\mathcal{O}(GN(\log GN)(\log N))\) time and requires \(\mathcal{O}(GN)\) space. We empirically show that our exact algorithm is _faster_ than Monte Carlo approximation techniques. The theory of our work is extensible to a more general class of test statistics which we discuss in App. B. We hope that this work encourages the use of exact paired-permutation tests in future NLP research.

### Ethical Concerns

We foresee no ethical concerns in this work.

\begin{table}
\begin{tabular}{l c c}
**Metric** & **Mean** & **Standard Dev.** \\ \hline Accuracy & \(0.9543\) & \(0.1116\) \\ Sentence length & \(12.08\) & \(10.60\) \\ \end{tabular}
\end{table}
Table 1: Distributions for of accuracy and sentence length for POS tagging using Stanza [20] on the English UD test dataset [14].

 



## Acknowledgments

We would like to thank the reviewers for their invaluable feedback and time spent engaging with our work. The first author is supported by the University of Cambridge School of Technology Vice-Chancellor's Scholarship as well as by the University of Cambridge Department of Computer Science and Technology's EPSRC.

## References

* [1]M. A. Abrams, K. Agilonova, L. Angrinov, M. J. Asahara, L. Ateyah, M. Attia, A. Atutxa, L. Batthasu, E. Badmaeva, M. Ballesteros, E. Banerjee, S. Bank, V. Barbuit, V. Bismkov, J. Chui, R. D. Davis, C. DeVille, P. Deshpande, G. DeVille, T. Dietterich, N. DeVille, H. de Vries, F. DeVille, P. Deshpande, G.



Alexei Lavrentiev, John Lee, Phuong Le Hong, Alessandro Lenci, Saran Leptradit, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, Kyung-Tae Lim, Nikola Ljubesic, Olga Loginova, Olga Nashevskaya, Teresa Lynn, Viven Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Catalina Maranduc, David Marecek, Katrin Marinheincke, Hector Martinez Alonso, Andre Martins, Jan Masek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonca, Niko Mieka, Margarita Misirpashayeva, Anna Missila, Catalin Mititela, Yusuke Miyao, Simonetta Montemagnari, Amir Moree, Laura Moreno Romero, Keiko Sophie Mori, Shinsuke Mori, Bjartur Mortensen, Bohdan Moskalexkyki, Kadri Muschnek, Yugo Murawaki, Kaili Muurisep, Pinkey Nainwani, Juan Ignacio Navarro Horniaeck, Anna Nedoluzhko, Gunta Nespore-Berkalane, Luong Nguyen Thi, Huyen Nguyen Thi Minh, Vitaly Nikolaev, Rattim Nitissaroj, Hanna Nurmi, Stina Ojala, Adedayo Diokun, Mai Omura, Petya Osenova, Robert Ostling, Lilja Ovrelid, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patelvik, Guilherme Paulino-Passos, Siyao Peng, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Pitluainen, Emily Pitler, Barbara Plank, Thierry Poibeau, Martin Popel, Lauma Pretkanina, Sophie Prevost, Prokpis PROKpidis, Adam Przepiorkowski, Tiina Puolakainen, Sampo Pyysalo, Andriela Raabis, Alexandre Rademaker, Loganathan Ramasamy, Taraka Rama, Carlos Ramisch, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Michael Riessler, Larissa Rinaldi, Laura Rietuma, Luisa Rocha, Mykhahio Romanenko, Rudolf Rosa, Davide Rovati, Valentin Rosca, Olga Rudina, Jack Rueter, Shoval Sadde, Benoit Sagot, Shadi Saleh, Tanja Samardzic, Stephanie Samson, Manuela Sanguinetti, Baiba Saulite, Yanin Sawanakunnanon, Nathan Schneider, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Moigana Seraji, Mo Shen, Atusko Shimada, Muhobhibssiri, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, Maria Simkova, Kiril Simov, Aaron Smith, Isabela Soares-Bastos, Carolyn Spadine, Antonio Stella, Milan Straka, Jana Strnadova, Alane Suhr, Umut Sulubacak, Zsolt Szanto, Dima Taji, Yuta Takahashi, Taakaishi Tanaka, Isabelle Tellier, Trond Trostendt, Anna Truhkina, Retut Tsafray, Francis Tyers, Sumire Uematsu, Zdefika Ursova, Larraitz Uria, Hans Uszkoreit, Sowmya Vajiala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemontte de la Clergerie, Veronika Vincze, Lars Wallin, Jing Xian Wang, Jonathan North Washington, Sey Williams, Mats Wirer, Tsegagw Woldermarian, Tak-sum Wong, Chunxiao Yan, Marat M. Yavrumyan, Zhuoran Yu, Zdenek Zaboktrtsky, Amir Zeldes, Daniel Zeman, Manying Zhang, and Hanzhi Zhu. 2018. Universal dependencies 2.3. LIMIT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* [Ojala and Garriga2010] Markus Ojala andemma C. Garriga. 2010. Permutation tests for studying classifier performance. _The Journal of Machine Learning Research_, 11:1833-1863.
* [Peyrard et al.2021] Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. 2021. Better than average: Paired evaluation of NLP systems. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLIJCNLP 2021, Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 2301-2315. Association for Computational Linguistics.
* [Qi et al.2020] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In _Proceedings of the Association for Computational Linguistics: System Demonstrations_.
* [Ross2008] Sheldon Ross. 2008. _A First Course in Probability_, 8th edition. Pearson.
* [Serlin2000] Ronald C. Serlin. 2000. Testing for robustness in Monte Carlo studies. _Psychological Methods_, 5(2):230.
* Yeh2000] Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In _COLING 2000 Volume 2: The 18th International Conference on Computational Linguistics_.

 

Proof of Correctness of convolve_DP

We prove the correctness of convolve_DP using the following lemma.

**Lemma 1**.: _For any \(N\) RVs \({\rm Z}_{1},...,{\rm Z}_{N}\) with PMFs \(f_{1},...,f_{N}\) respectively and \(n\in\{1,...,N\}\), convolve_DP\((f_{1},...,f_{N})\) constructs \({\rm F}_{n}\) such that for any \(\xi\in\mathcal{S}\),_

\[{\rm F}_{n}(\xi)=f_{:n}(\xi)\stackrel{{\text{\tiny def}}}{{=}}(f_{ 1}\star\cdots\star f_{n})(\xi) \tag{10}\]

Proof.: We prove this by induction on \(N\).

_Base case: \(N=1\)._ We have that \({\rm F}_{0}(0)=1\) and \({\rm F}_{0}(\xi)=0\) for all \(\xi\in\mathcal{S}\smallsetminus\{0\}\). Therefore, \({\rm F}_{1}(\overrightarrow{\xi_{n}})={\rm F}_{1}(\overleftarrow{\xi_{n}})= \frac{1}{2}\) and \({\rm F}_{1}(\xi)=0\) for all \(\xi\in\mathcal{S}\smallsetminus\{\overrightarrow{\xi_{n}},\overleftarrow{\xi_{ n}}\}\) as expected.

_Inductive step:_ Assume (10) holds for \(N=n-1\). Let \(N=n\) and consider \(f_{:(n-1)}\star f_{n}\).

\[(f_{:(n-1)}\star f_{n})(\xi)=\sum_{\xi^{\prime}\in\mathcal{S}}\!\!f_{:(n-1)}( \xi^{\prime})f_{n}(\xi-\xi^{\prime})=\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \! 

We can therefore use four-additively decomposable functions, \({\mathbf{g_{1}}}\) to \({\mathbf{g_{4}}}\), that decompose such that \(g_{1}(u_{n},v_{n})=g_{3}(v_{n},u_{n})=\operatorname{tp}(u_{n})\) and \(g_{2}(u_{n},v_{n})=g_{4}(v_{n},u_{n})=\operatorname{in}(u_{n})\). Our \(h\) function then takes four arguments and can be defined as

\[h(x_{1},x_{2},x_{3},x_{4})\stackrel{{\text{def}}}{{=}}\frac{x_{1 }}{x_{1}+\tfrac{1}{2}x_{2}}-\frac{x_{3}}{x_{3}+\tfrac{1}{2}x_{4}} \tag{16}\]

We can additionally apply an absolute value to \(h\) to check for the absolute difference in \(F_{1}\) scores; doing this would make the significance test two-tailed rather than one-tailed.