SlowGuess/NAACL_2022_OCR huggingface dataset:
4The experiment used an Apple M1 Max processor.
<sup>5</sup>We note that the average test set size of the 129 UD treebanks we examined is just over 1000 sentences, and only three treebanks had more than 6000 sentences.
We would like to thank the reviewers for their invaluable feedback and time spent engaging with our work. The first author is supported by the University of Cambridge School of Technology Vice-Chancellor's Scholarship as well as by the University of Cambridge Department of Computer Science and Technology's EPSRC.
Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL 2012, July 12-14, 2012, Jeju Island, Korea, pages 995-1005. ACL.
Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176-181, Portland, Oregon, USA. Association for Computational Linguistics.
James W. Cooley and John W. Tukey. 1965. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation, 19(90):297-301.
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2022. Introduction to Algorithms, 4 edition. MIT Press.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132-1146.
Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. *Neural Computation*, 10(7):1895-1923.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Statistical Significance Testing for Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
Phillip Good. 2000. Permutation Tests A Practical Guide to Resampling Methods for Testing Hypotheses. Springer.
Elizabeth Koehler, Elizabeth Brown, and Sebastien J. Haneuse. 2009. On the assessment of Monte Carlo error in simulation-based statistical analyses. The American Statistician, 63(2):155-162.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Erich Leo Lehmann and Joseph P. Romano. 2005. Testing Statistical Hypotheses. Springer.
Steven S. Muchnick. 1998. Advanced Compiler Design and Implementation. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Victoria Basmov, John Bauer, Sandra Bellato, Kepa Bengoetxea, Yevgeni Berzak, Irshad Ahmad Bhat, Riyadh Ahmad Bhat, Erica Biagetti, Eckhard Bick, Rogier Blokland, Victoria Bobicev, Carl Borstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Gülsen Cebiroğlu Eryigit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Jayeol Chun, Silvie Cinková, Aurélie Collomb, Çagri Öltekin, Miriam Connor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Díaz de Ilarraza, Carly Dickerson, Peter Dirix, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Tomáž Erjavec, Aline Etienne, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Katarína Gajdoshová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Kim Gerdes, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matías Grioni, Normunds Grūzītis, Bruno Guillaume, Céline Guillot-Barbance, Nizar Habash, Jan Hajic, Jan Hajic jr., Linh Hà My, Na-Rae Han, Kim Harris, Dag Haug, Barbora Hladka, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Jena Hwang, Radu Ion, Elena Irimia, Olajíde Ishola, Tomáš Jelinek, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşikara, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Kamil Kopacewicz, Natalia Kotsyba, Simon Krek, Sookyoung Kwak, Veronika Laippala, Lorenzo Lambertino, Lucia Lam, Tatiana Lando, Septina Dian Larasati
Alexei Lavrentiev, John Lee, Phuong Lê Hùng, Alessandro Lenci, Saran Lertpradit, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, Kyung-Tae Lim, Nikola Ljubesic, Olga Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Catalina Maranduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonça, Niko Miekka, Margarita Misirpashayeva, Anna Missilä, Catalin Mititelu, Yusuke Miyao, Simonetta Montemagni, Amir More, Laura Moreno Romero, Keiko Sophie Mori, Shinsuke Mori, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Juan Ignacio Navarro Horniacek, Anna Nedoluzhko, Gunta Nespore-Berzkalne, Luong Nguyen Thi, Huyen Nguyen Thi Minh, Vitaly Nikolaev, Rattima Nitisaroj, Hanna Nurmi, Stina Ojala, Adedayo Olókun, Mai Omura, Petya Osenova, Robert Östling, Lilja Øvrelid, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme PaulinoPassos, Siyao Peng, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Emily Pitler, Barbara Plank, Thierry Poibeau, Martin Popel, Lauma Pretkalnina, Sophie Prevost, Prokopis Prokopidis, Adam Przepiörkowski, Tiina Puolakainen, Sampo Pyysalo, Andriela Räabis, Alexandre Rademaker, Loganathan Ramasamy, Taraka Rama, Carlos Ramisch, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Michael Rießler, Larissa Rinaldi, Laura Rituma, Luisa Rocha, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Valentin Rosca, Olga Rudina, Jack Rueter, Shoval Sadde, Benoit Sagot, Shadi Saleh, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Baiba Saulite, Yanin Sawanakunanon, Nathan Schneider, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen Atsuko Shimada, Muh Shohibussirri, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu Katalin Simko, María Simković, Kiril Simov, Aaron Smith Isabela Soares-Bastos Carolyn Spadine Antonio Stella Milan Straka Jana Strnadová Alane Suhr Umut Sulubacak Zsolt Szántó Dima Taji Yuta Takahashi Takaaki Tanaka Isabelle Tellier Trond Trosterud Anna Trukhina Reut Tsarfaty Francis Tyers Sumire Uematsu Zdenka Urešová Larraitz Uria Hans Uszkoreit Sowmya Vajjala Daniel van Niekerk Gertjan van Noord Viktor Varga Eric Villemonte de la Clergerie Veronika Vincze Lars Wallin Jing Xian Wang Jonathan North Washington Seyi Williams Mats Wiren Tsegay Woldemariam Tak-sum Wong Chunxiao Yan Marat M. Yavrumyan Zhuoran Yu Zdeněk Žabokrtský Amir Zeldes Daniel Zeman Manying Zhang and Hanzhi Zhu. 2018. Universal dependencies 2.3. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics Charles University.
Markus Ojala and Gemma C. Garriga. 2010. Permutation tests for studying classifier performance. The
Journal of Machine Learning Research, 11:1833-1863.
Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. 2021. Better than average: Paired evaluation of NLP systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2301-2315. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the Association for Computational Linguistics: System Demonstrations.
Sheldon Ross. 2008. A First Course in Probability, 8th edition. Pearson.
Ronald C. Serlin. 2000. Testing for robustness in Monte Carlo studies. Psychological Methods, 5(2):230.
Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In *COLING* 2000 Volume 2: The 18th International Conference on Computational Linguistics.
A Proof of Correctness of convolve_DP
We prove the correctness of convolve_DP using the following lemma.
Lemma 1. For any  $N$  RVs  $\mathbf{Z}_1, \dots, \mathbf{Z}_N$  with PMFs  $f_1, \dots, f_N$  respectively and  $n \in \{1, \dots, N\}$ , convolve  $\mathsf{DP}(f_1, \dots, f_N)$  constructs  $\mathbf{F}_n$  such that for any  $\xi \in S$ ,
$$
\mathrm {F} _ {n} (\xi) = f _ {: n} (\xi) \stackrel {\text {d e f}} {=} \left(f _ {1} \star \dots \star f _ {n}\right) (\xi) \tag {10}
$$
Proof. We prove this by induction on  $N$ .
Base case:  $N = 1$ . We have that  $\mathrm{F}_0(0) = 1$  and  $\mathrm{F}_0(\xi) = 0$  for all  $\xi \in S \setminus \{0\}$ . Therefore,  $\mathrm{F}_1(\overrightarrow{\xi_n}) = \mathrm{F}_1(\overleftarrow{\xi_n}) = \frac{1}{2}$  and  $\mathrm{F}_1(\xi) = 0$  for all  $\xi \in S \setminus \{\overrightarrow{\xi_n}, \overleftarrow{\xi_n}\}$  as expected.
Inductive step: Assume (10) holds for  $N = n - 1$ . Let  $N = n$  and consider  $f_{:(n - 1)} \star f_n$ .
$$
(f _ {: (n - 1)} \star f _ {n}) (\xi) = \sum_ {\xi^ {\prime} \in \mathcal {S}} f _ {: (n - 1)} \left(\xi^ {\prime}\right) f _ {n} \left(\xi - \xi^ {\prime}\right) = \sum_ {\xi^ {\prime} \in \operatorname {d o m} \left(f _ {n}\right)} f _ {n} \left(\xi^ {\prime}\right) = \sum_ {\xi^ {\prime} \in \operatorname {d o m} \left(f _ {n}\right)} \mathrm {F} _ {n - 1} \left(\xi + \xi^ {\prime}\right) f _ {n} \left(\xi^ {\prime}\right) \tag {11}
$$
This is exactly the construction in the for-loop between Line 7 and Line 8. Therefore,  $\mathrm{F}_n(\xi) = f_{:n}(\xi)$ .
As  $\mathrm{F}_N$  will contain the  $N$ -fold convolution  $f_1 \star \dots \star f_N$ , convolve_DP is correct by definition.
B Paired-Permutation Test for Higher-order Test Statistics
In this section, we extend our approach for the paired-permutation test to test statistics that are functions of  $m$  additively decomposable functions. In symbols, this assumption means that we can rewrite  $t$  as follows
$$
t (\mathbf {u}, \mathbf {v}) \stackrel {\text {d e f}} {=} h \left(\mathbf {g} _ {1} (\mathbf {u}, \mathbf {v}), \dots , \mathbf {g} _ {m} (\mathbf {u}, \mathbf {v})\right) \tag {12}
$$
for any function  $h$  and integer-valued, additive decomposable functions  $g_{i}$  (i.e.,  $g_{i}(\mathbf{u},\mathbf{v}) \stackrel{\mathrm{def}}{=} \sum_{n=1}^{N} g_{i}(u_{n},v_{n})$ ). We now define  $\vec{\xi_{n}}$  and  $\vec{\xi_{n}}$  as  $m$ -tuples,
$$
\xrightarrow {\rightarrow} \xi_ {n} \stackrel {\text {d e f}} {=} \langle g _ {1} (\mathbf {u}, \mathbf {v}), \dots , g _ {m} (\mathbf {u}, \mathbf {v}) \rangle \tag {13}
$$
$$
\xleftarrow {\xi_ {n}} \stackrel {\text {d e f}} {=} \langle \mathbf {g} _ {1} (\mathbf {v}, \mathbf {u}), \dots , \mathbf {g} _ {m} (\mathbf {v}, \mathbf {u}) \rangle \tag {14}
$$
And so each RV  $Z_{n}$  has the same PMF as in (4b). We can then define an analogous function to exact_perm_test for the case of  $m$  additively decomposable functions. We give pseudocode for this as exact_perm_test in Alg 5. The convolution algorithms, convolve_DP and convolve_FFT, can both be used to perform for convolution step in Line 7.
Theorem 5. For any two entries,  $\mathbf{u}$  and  $\mathbf{v}$ , and test statistic  $t$  that factorizes as in (12) with  $h$  and  $g_{1}$  to  $g_{m}$ , exact_perm_test $_{\mathfrak{m}}(\mathbf{u}, \mathbf{v}, g_{1}, \dots, g_{m}, h)$  returns  $p$  in  $\mathcal{O}(G^{m}N^{m}(\log GN)(\log N))$  time and  $\mathcal{O}(G^{m}N^{m}\log N)$  space.
Proof. The proof of correctness for exact_perm_test $_{\mathfrak{m}}$  is the same as Theorem 2. The expensive operation in the algorithm is the convolution step (Line 7). We can perform a FFT  $m$ -dimensional convolution in  $\mathcal{O}(G^m N^m \log GN)$  time and  $\mathcal{O}(G^m N^m)$  space. As we require  $\mathcal{O}(\log N)$  convolution steps, exact_perm_test $_{\mathfrak{m}}$  runs in  $\mathcal{O}(G^m N^m (\log GN)(\log N))$  time and  $\mathcal{O}(G^m N^m \log N)$  space.
Example. A common example for a test statistic that requires multiple additively decomposable functions is the difference in  $F_{1}$  scores. Similar to accuracy, each entry  $u_{n} \in \{1,\dots,C\}^{G}$  and  $G$  is the maximum length of an entry sequence. Let  $\mathrm{tp}(u_n)$  and  $\mathrm{in}(u_n)$  be the number of true positive and incorrect predictions made in entry  $u_{n}$  respectively. Then the difference in  $F_{1}$  scores can be given as
$$
t (\mathbf {u}, \mathbf {v}) \stackrel {\text {d e f}} {=} \frac {\sum_ {n = 1} ^ {N} \operatorname {t p} \left(u _ {n}\right)}{\sum_ {n = 1} ^ {N} \operatorname {t p} \left(u _ {n}\right) + \frac {1}{2} \sum_ {n = 1} ^ {N} \operatorname {i n} \left(u _ {n}\right)} - \frac {\sum_ {n = 1} ^ {N} \operatorname {t p} \left(v _ {n}\right)}{\sum_ {n = 1} ^ {N} \operatorname {t p} \left(v _ {n}\right) + \frac {1}{2} \sum_ {n = 1} ^ {N} \operatorname {i n} \left(v _ {n}\right)} \tag {15}
$$
1: def exactperm_test  $\mathfrak{m}(\mathbf{u},\mathbf{v},g_1,\dots,g_m,h)$  ..
2: for  $n = 1$  to  $N$ :
3:  $\overrightarrow{\xi_n} \gets \langle g_1(u_n, v_n), \dots, g_m(u_n, v_n) \rangle$
4:  $\stackrel{\leftarrow}{\xi_n} \gets \langle g_1(v_n, u_n), \dots, g_m(v_n, u_n) \rangle$
5:  $f_{n}\gets \mathrm{pmf}_{\mathrm{Z}}\left(\overrightarrow{\xi_{n}},\overleftarrow{\xi_{n}}\right)$
6:  $\overline{\xi} \gets h(\pmb{g}_1(\mathbf{u},\mathbf{v}),\dots,\pmb{g}_2(\mathbf{u},\mathbf{v}))$
7:  $f_{\mathrm{S}} \gets f_1 \star \dots \star f_N$
8:  $\triangleright$  Sum-up the PMF to get  $p$
9: return  $\sum_{\xi \in \mathcal{S}}f_{\mathrm{S}}(\xi)$  1  $\mathbb{1}\left[h(\xi)\geq \overline{\xi}\right]$
$\triangleright$  Local effect (stay)
$\triangleright$  Local effect (swap)
$\triangleright$  Compute observed effect
$\triangleright$  Convolve the  $f_{n}$  's
Algorithm 5: Pseudocode to find exact  $p$  value for the paired-permutation test for test statistics comprised of  $m$  additively decomposable functions.
We can therefore use four-additively decomposable functions,  $g_{1}$  to  $g_{4}$ , that decompose such that  $g_{1}(u_{n}, v_{n}) = g_{3}(v_{n}, u_{n}) = \mathrm{tp}(u_{n})$  and  $g_{2}(u_{n}, v_{n}) = g_{4}(v_{n}, u_{n}) = \mathrm{in}(u_{n})$ . Our  $h$  function then takes four arguments and can be defined as
$$
h \left(x _ {1}, x _ {2}, x _ {3}, x _ {4}\right) \stackrel {\text {d e f}} {=} \frac {x _ {1}}{x _ {1} + \frac {1}{2} x _ {2}} - \frac {x _ {3}}{x _ {3} + \frac {1}{2} x _ {4}} \tag {16}
$$
We can additionally apply an absolute value to  $h$  to check for the absolute difference in  $F_{1}$  scores; doing this would make the significance test two-tailed rather than one-tailed.