SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As discussed in Section 6, our instead work assumes that all outcomes are observable and, importantly, that they are not affected by the human predictions. We also do not explicitly grapple with whether the human expert has an objective other than maximizing accuracy under a known metric (e.g., squared error). Though this is often a primary concern in many high-stakes settings – for example, ensuring that bail decisions are not only accurate but also nondiscriminatory – it is outside the scope of our work, and we refer the reader instead to Rambachan (2022) for further discussion.
As discussed in section 1, another closely related theme is directly comparing human performance to that of an algorithm (Cowgill (2018), Dawes et al. (1989), Grove et al. (2000)), and developing learning algorithms which are complementary to human expertise (Madras et al. (2018), Raghu et al. (2019), Mozannar and Sontag (2020), Keswani et al. (2021), Agrawal et al. (2018) and Bastani et al. (2021)). A key design consideration when designing algorithms to complement human expertise involves reasoning about the ways in which humans may respond to the introduction of an algorithm, which may be strategic (e.g. Kleinberg and Raghavan (2018), Perdomo et al. (2020), Cen and Shah (2021), Hardt et al. (2015), Liu et al. (2020)) or subject to behavioral biases (Kleinberg et al. (2022)). These behaviors can make it challenging to design algorithms which work with humans to achieve the desired outcomes, as humans may respond to algorithmic recommendations or feedback in unpredictable ways.
Conditional independence testing. We cast our setting as a special case of conditional independence testing, which has been well studied in the statistics community. For background we refer the reader to Dawid (1979). It has long been known that testing conditional independence between three (possibly high-dimensional) random variables is a challenging problem, and the recent result of Shah and Peters (2018) demonstrates that this is in fact impossible in full generality. Nonetheless, there are many methods for testing conditional independence under natural assumptions; perhaps the most popular are the kernel-based methods introduced by Fukumizu et al. (2004) and subsequently developed in Gretton et al. (2007) and Zhang et al. (2011), among others.
Our work instead takes inspiration from the 'knockoffs' framework developed in Candès et al. (2016), Barber et al. (2018) and Barber and Candès (2019), as well as the closely related conditional permutation test of Berrett et al. (2018). These works leverage the elementary observation that, under the null hypothesis that (specialized to our notation) the outcome  $Y$  and prediction  $\hat{Y}$  are independent conditional on the observed data  $X$ , new samples from the distribution of  $\hat{Y} \mid X$  should be exchangeable with  $\hat{Y}$ . Thus, if we know - or can accurately estimate - the distribution of  $\hat{Y} \mid X$ , it is straightforward to generate fresh samples ('knockoffs') which are statistically indistinguishable from the original data under the null hypothesis  $H_0: Y \perp \hat{Y} \mid X$ . Thus, if the observed data appears anomalous with respect to these knockoffs, this may provide us a basis on which to reject  $H_0$ .
Our work avoids taking inspiration from this framework, but avoids estimating the distribution of  $\hat{Y} \mid X$  by instead leveraging a simple nearest-neighbors style algorithm for generating knockoffs. In that sense, our technique builds upon the nearest-neighbors based estimator of Runge (2017), and is
nearly identical to the one-nearest-neighbor procedure proposed in the 'model-powered' conditional independence test of Sen et al. (2017). This algorithm is a subroutine in their more complicated end-to-end procedure, which involves training a model to distinguish between the observed data and knockoffs generated via swapping the 'predictions' (again specializing their general test to our setting) associated with instances which are as close as possible under the  $\ell_2$  norm. By contrast, we analyze a similar procedure under different smoothness assumptions which allow us to recover p-values that are entirely model free.
We establish the proof of Theorem 1 following the intuition presented in Section 3. Specifically, we first bound the type I error of ExpertTest in the idealized case where the data set contains  $L$  identical pairs of observations  $x = x'$ . We then refine this bound to handle the case, which is more likely in practice, that the pairs chosen are merely close together. Our final bound thus includes additional approximation error to account for the 'similarity' of the pairs – if we succeed in finding  $L$  pairs which are identical, we get nearly exact type I error control, whereas if we are forced to pair instances which are 'far apart', we incur additional approximation error. We formalize this intuition below.
An idealized bound. We first establish that  $\mathbb{P}(\tau_K\leq \alpha)\leq \alpha +\frac{1}{K + 1}$  for any  $\alpha \in [0,1]$  when  $x = x^{\prime}$  for every  $(x,x^{\prime})$  pair chosen by ExpertTest.
To that end, we observe  $n$  data points  $(x_{i},y_{i},\hat{y}_{i})$ ,  $i \in [n]$ . Let  $\mathcal{L} = \{i_{2\ell -1},i_{2\ell} : \ell \in [L]\}$  denote the indices of the pairs chosen by ExpertTest, with  $(x_{i_{2\ell -1}},x_{i_{2\ell}})$  for  $\ell \in [L]$  denoting the pairs themselves.
By assumption, ExpertTest succeeds in finding identical pairs:
$$
x _ {i _ {2 \ell - 1}} = x _ {i _ {2 \ell}}, \forall \ell \in [ L ]. \tag {14}
$$
Therefore, from the definition (9) it follows that  $r((x_{i_{2\ell -1}},\hat{y}_{i_{2\ell -1}}),(x_{i_{2\ell}},\hat{y}_{i_{2\ell}})) = 1$  for all  $\ell \in [L]$ .
As discussed in Section 3, ExpertTest will repeatedly generate  $n$  fresh data points, denoted by  $\tilde{D}$ , as follows. For each index  $i \in [n] \backslash \mathcal{L}$ , i.e. those not corresponding to those selected in  $L$  pairs, we select exactly the observed data  $(x_i, y_i, \hat{y}_i)$ .
For  $i \in \mathcal{L}$ , we sample a data triplet as follows: for  $i \in \{i_{2\ell-1}, i_{2\ell}\}$ , we let  $(x_{i2\ell-1}, y_{i2\ell-1})$ ,  $(x_{i2\ell}, y_{i2\ell})$  be the observed values but sample the corresponding  $\hat{y}$  values from  $\{(\hat{y}_{2\ell-1}, \hat{y}_{2\ell}), (\hat{y}_{2\ell}, \hat{y}_{2\ell-1})\}$  with equal probability. That is, we swap the  $\hat{y}$  values associated with  $(x_{i2\ell-1}, y_{i2\ell-1})$ ,  $(x_{i2\ell}, y_{i2\ell})$  with probability  $\frac{1}{2}$ . We argue that this resampling process is implicitly generating a fresh, identically distributed dataset from the underlying distribution  $\mathcal{D}$  conditioned on the following event  $\mathcal{F}$ :
$$
\mathcal {F} = \left\{\left(x _ {i}, y _ {i}, \hat {y} _ {i}\right): i \in [ n ] \backslash \mathcal {L} \right\} \cup \left\{\left(x _ {i}, y _ {i}\right): i \in \mathcal {L} \right\} \cup \left\{\left(\hat {y} _ {i _ {2 \ell - 1}}, \hat {y} _ {i _ {2 \ell}}\right) \vee \left(\hat {y} _ {i _ {2 \ell}}, \hat {y} _ {i _ {2 \ell - 1}}\right): \ell \in [ L ] \right\}. \tag {15}
$$
Why condition on  $\mathcal{F}$ ? As discussed in section 3, a straightforward test would involve simply resampling  $K$  fresh datasets from the underlying distribution  $\mathcal{D}_X \times \mathcal{D}_{\hat{Y} |X} \times \mathcal{D}_{Y|X}$  and observing that, by definition, these datasets are distributed identically to the observed data  $D_0$  under  $H_0: Y \perp \hat{Y} \mid X$ . While this would form the basis for a valid test along the lines of the one described in Section 3, it requires knowledge of the underlying distribution which we are unlikely to have in practice. Thus, we instead condition on nearly everything in the observed data – the values and exact ordering of  $X$  and the values and exact ordering of  $Y$ , and the values of  $\hat{Y}$  up to a specific set of allowed permutations (those induced by swapping 0 or more paired  $\hat{y}_{i_{2\ell-1}}, \hat{y}_{i_{2\ell}}$  values). This substantially simplifies the resampling problem, as we only need to reason about the correct 'swap' probability for each such pair. This can be viewed as an alternative factorization of the underlying distribution  $\mathcal{D}$  under  $H_0$  – rather than sampling  $X \sim \mathcal{D}_X, Y \sim \mathcal{D}_{Y|X}, \hat{Y} \sim \mathcal{D}_{\hat{Y} |X}$ , instead sample an event  $\mathcal{F} \sim \mathcal{D}_{\mathcal{F}}$  from the induced distribution over events of the form (15), and then sample  $\hat{Y} \sim \mathcal{D}_{\hat{Y} | \mathcal{F}}$ .
First, we show that conditional on  $\mathcal{F}$ , the resampled dataset  $\tilde{D}$  and the observed dataset  $D_0$  are indeed identically distributed under  $H_0: Y \perp \hat{Y} \mid X$  (that they are also independent, conditional on  $\mathcal{F}$ , is
clear by construction). To see this, observe that for each  $\ell \in [L]$ :
$$
\begin{array}{l} \mathbb {P} \left(\left(x _ {i _ {2 \ell - 1}}, y _ {i _ {2 \ell - 1}}, \hat {y} _ {i _ {2 \ell - 1}}\right), \left(x _ {i _ {2 \ell}}, y _ {i _ {2 \ell}}, \hat {y} _ {i _ {2 \ell}}\right)\right) (16) \\ = \mathbb {P} \left(x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(y _ {i _ {2 \ell - 1}} \mid x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(\hat {y} _ {i _ {2 \ell - 1}} \mid x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(x _ {i _ {2 \ell}}\right) \mathbb {P} \left(y _ {i _ {2 \ell}} \mid x _ {i _ {2 \ell}}\right) \mathbb {P} \left(\hat {y} _ {i _ {2 \ell}} \mid x _ {i _ {2 \ell}}\right) (17) \\ = \mathbb {P} \left(x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(y _ {i _ {2 \ell - 1}} \mid x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(\hat {y} _ {i _ {2 \ell}} \mid x _ {i _ {2 \ell - 1}}\right) \mathbb {P} \left(x _ {i _ {2 \ell}}\right) \mathbb {P} \left(y _ {i _ {2 \ell}} \mid x _ {i _ {2 \ell}}\right) \mathbb {P} \left(\hat {y} _ {i _ {2 \ell - 1}} \mid x _ {i _ {2 \ell}}\right) (18) \\ = \mathbb {P} \left(\left(x _ {i _ {2 \ell - 1}}, y _ {i _ {2 \ell - 1}}, \hat {y} _ {i _ {2 \ell}}\right), \left(x _ {i _ {2 \ell}}, y _ {i _ {2 \ell}}, \hat {y} _ {i _ {2 \ell - 1}}\right)\right) (19) \\ \end{array}
$$
In above, (17) follows from  $H_0$  and the assumption that the data are drawn i.i.d., and (18) follows from assumption (14) that  $x_{i_{2\ell - 1}} = x_{i_{2\ell}}$ . By construction, the events in (16) and (19) are the only two possible outcomes after conditioning on  $\mathcal{F}$ , and this simple argument shows that in fact they are equally likely.
Thus, let  $\tilde{D}_1,\ldots ,\tilde{D}_K$  be  $K$  independent and identically distributed datasets generated by the above procedure. Let  $\tilde{D}_0$  be one additional sample from this distribution, which we showed was distributed identically to  $D_0$  under the idealized assumption (14).
As discussed in Section 3, for any real-valued function  $F$  that maps each dataset to  $\mathbb{R}$ , we have
$$
\tau_ {K} = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {1} [ F (\tilde {D} _ {0}) \lesssim F (\tilde {D} _ {k}) ] \tag {20}
$$
where we use definition of  $\mathbb{1}[\cdot \lesssim \cdot ]$  as in (7).
Because  $\tilde{D}_0, \ldots, \tilde{D}_K$  are i.i.d., and thus exchangeable, it follows that  $\frac{1}{K} \sum_{k=1}^{K} \mathbb{1}[F(\tilde{D}_0) \lesssim F(\tilde{D}_k]$  is uniformly distributed  $\{0, \frac{1}{K}, \ldots, 1\}$ . Therefore, with a little algebra it can be verified that for any  $\alpha \in [0,1]$ ,  $\tau_K$  satisfies
$$
\mathbb {P} _ {\tilde {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}} (\tau_ {K} \leq \alpha) \leq \alpha + \frac {1}{K + 1}. \tag {21}
$$
Because  $D_0$  and  $\tilde{D}_0$  are independent and identically distributed under (14), the same holds if we replace  $\tilde{D}_0$  with  $D_0$ . Thus, ExpertTest provides nearly exact type I error control in the case that the idealized assumption (14) holds. This result will serve as a useful building block, as we'll now proceed to relax this assumption and bound the type I error of ExpertTest in terms of the total variation distance between  $\tilde{D}_0$  and  $D_0$ .
Fixing the approximation.  $\tilde{D}_1, \ldots, \tilde{D}_K$  are synthetically generated datasets that are independent and identically distributed. The argument above replaced the observed dataset  $D_0$  with a resampled 'idealized' dataset  $\tilde{D}_0$ , which is also independent and identically distributed with respect to  $\tilde{D}_1, \ldots, \tilde{D}_K$ , and then used this fact to demonstrate that  $\mathbb{P}_{\tilde{D}_0, \ldots, \tilde{D}_K | \mathcal{F}}(\tau \leq \alpha) \leq \alpha + \frac{1}{K + 1}$ . If the idealized assumption (14) holds, replacing  $D_0$  with  $\tilde{D}_0$  is immaterial as we showed the two are identically distributed conditional on  $\mathcal{F}$ . Of course, this assumption will not hold in general, and this is what we seek to correct next.
Let  $\bar{D}_0\sim \mathcal{D}_{\cdot |\mathcal{F}}$  be a random variable distributed according to the true underlying distribution  $\mathcal{D}$ , conditional on the event  $\mathcal{F}$ . The observed data  $D_0$  can be interpreted as one realization of this random variable. One way to quantify the excess type I error incurred by using  $\tilde{D}_0$  in place of  $D_0$  is to bound the total variation distance between the joint distributions of  $(\tilde{D}_0,\dots \tilde{D}_K)$  and that of  $(\bar{D}_0,\tilde{D}_1,\dots \tilde{D}_K)$ . Specifically, it follows from the definition of total variation distance that:
$$
\mathbb {P} _ {\bar {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}} (\tau_ {K} \leq \alpha) \leq \mathbb {P} _ {\tilde {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}} (\tau_ {K} \leq \alpha) + \mathrm {T V} \left(\mathbb {P} _ {\bar {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}}, \mathbb {P} _ {\tilde {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}}\right), \tag {22}
$$
where  $\mathrm{TV}(\mathbb{P}_{\tilde{D}_0,\dots,\tilde{D}_K|F},\mathbb{P}_{\tilde{D}_0,\dots,\tilde{D}_K|F})$  denotes the total variation distance between its arguments. Due to the independence of the resampled datasets, this simplifies to:
$$
\mathsf {T V} \left(\mathbb {P} _ {\tilde {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}}, \mathbb {P} _ {\tilde {D} _ {0}, \dots , \tilde {D} _ {K} | \mathcal {F}}\right) = \mathsf {T V} \left(\mathbb {P} _ {\bar {D} _ {0} | \mathcal {F}}, \mathbb {P} _ {\tilde {D} _ {0} | \mathcal {F}}\right). \tag {23}
$$
Therefore, we need only bound the total variation distance between  $\mathbb{P}_{\bar{D}_0|\mathcal{F}}$  and  $\mathbb{P}_{\tilde{D}_0|\mathcal{F}}$  to conclude the proof.
As defined in (11), the  $\varepsilon_{n,L}^{*}$  provides us with a way of bounding the total variation distance between the distribution of  $\bar{D}_0$  and  $\tilde{D}_0$ . To see this, observe that the distributions of  $\tilde{D}_0$  and  $\bar{D}_0$ , conditioned on  $\mathcal{F}$ , can be described as follows. To construct  $\tilde{D}_0$ , we can imagine flipping  $L$  fair coins to decide the assignment of  $\hat{y}_i$  in each of the  $(\hat{y}_{i_{2\ell -2}}\hat{y}_{i_{2\ell}})$  pairs; if it comes up heads, we swap the observed pair  $(\hat{y}_{i_{2\ell -2}}\hat{y}_{i_{2\ell}})$  and if it comes up tails we do not. The observed  $(x_i,y_i)$  as well as  $\hat{y}_i$  for  $i\notin \mathcal{L}$  are set in  $\tilde{D}_0$  as they are observed in  $D_0$ .
$\bar{D}_0$  is constructed similarly, but we instead flip a coin with bias  $(1 + r((x_{i_{2\ell -1}},\hat{y}_{i_{2\ell -1}}),(x_{i_{2\ell}},\hat{y}_{i_{2\ell}})))^{-1}$  to decide the assignment of  $(\hat{y}_{i_{2\ell -2}}\hat{y}_{i_{2\ell}})$  - again, heads indicates that we swap the observed ordering, and tails indicate that we do not.
By construction, the distributions of  $\bar{D}_0$  and  $D_0$  are identical conditioned on  $\mathcal{F}$ , as  $r((x_{i_{2\ell -1}},\hat{y}_{i_{2\ell -1}}),(x_{i_{2\ell}},\hat{y}_{i_{2\ell}}))$  denotes the true relative odds of observing each of the two possible  $(x,\hat{y})$  pairings. In contrast, the distribution of  $\tilde{D}_0$  is different, as it was sampled using the simplifying assumption (14) - in particular,  $\tilde{D}_0$  is generated assuming  $r((x_{i_{2\ell -1}},\hat{y}_{i_{2\ell -1}}),(x_{i_{2\ell}},\hat{y}_{i_{2\ell}})) = 1!$
The difference between the biases of these coins is bounded above by  $\varepsilon_{n,L}^{*}$ . We'll use this observation, along with the following lemma, to complete the proof.
Lemma 3 (Bounding the total variation distance between i.i.d. coin flips) Let  $i \in [L]$  index a sequence of i.i.d. coin flips  $u_{1} \ldots u_{L}$  each with bias  $p_{i}$ , and  $v_{1} \ldots v_{L}$  be a sequence of i.i.d. coin flips with bias  $q_{i}$ . Then we can show:
$$
\operatorname {T V} \left(\left(u _ {1} \dots u _ {L}\right), \left(v _ {1} \dots v _ {L}\right)\right) \leq 1 - \left(1 - \max  _ {i} \left| p _ {i} - q _ {i} \right|\right) ^ {L} \tag {24}
$$
We defer the proof of lemma 3 to Appendix D. This implies that the total variation distance between  $\bar{D}_0$  and  $\tilde{D}_0$  is bounded above by  $1 - (1 - \varepsilon_{n,L}^*)^L$ . This, along with (21), (22) and (23) concludes the proof of Theorem 1.
Corollary 3.1 (Weaker type I error bound)
$$
\mathbb {P} (\tau \leq \alpha) \leq \alpha + \varepsilon_ {n, L} ^ {*} L + \frac {1}{K + 1} \tag {25}
$$
Corollary 3.1 is a weaker bound than the one given in Theorem 1, but is easier to interpret and manipulate. We will make use of this fact in the following section; the proof is an immediate consequence of theorem 1 and provided in Appendix D for completeness.
To establish theorem 2, we will argue that  $\varepsilon_{n,L}^{*}$  goes to 0 at a rate of  $O(n^{-\frac{1}{d}})$ . This implies that, provided  $L = o(n^{\frac{1}{d}})$ , the excess type I error established in theorem 1 is  $o(1)$  as desired. To do this, we first show that each pair  $(x_{i_{2\ell -1}}, x_{i_{2\ell}})$  chosen by ExpertTest will be close under the  $\ell_2$  norm (lemmas 4 and 5 below). We then leverage the smoothness assumption (12) to demonstrate that this further implies that  $\varepsilon_{n,L}^{*}$  concentrates around 0. For clarity we state auxiliary lemmas inline, and defer proofs to Appendix D.
Finding pairs which are close under the  $\ell_2$  norm.
Let  $M_L$  to be the set of matchings of size  $L$  on  $x_1 \ldots x_n$ ; i.e. each element of  $M_L$  is a set of  $L$  disjoint  $(x, x')$  pairs. Let  $m_L^*$  be the 'optimal' matching satisfying:
$$
m _ {L} ^ {*} \in \operatorname * {a r g   m i n} _ {z \in M _ {L}} \max  _ {(x, x ^ {\prime}) \in z} \| x - x ^ {\prime} \| _ {2}. \tag {26}
$$
<sup>6</sup>This technique is inspired by the proof of type I error control given for the Conditional Permutation Test in Berrett et al. (2018); see Appendix A.2 of their work for details
That is,  $m_{L}^{*}$  minimizes the maximum distance between any pair of observations in a mutually disjoint pairing of  $2L$  observations. Let
$$
d _ {L} ^ {*} = \max  _ {(x, x ^ {\prime}) \in m _ {L} ^ {*}} \| x - x ^ {\prime} \| _ {2}. \tag {27}
$$
That is, the smallest achievable maximum  $\ell_2$  distance over all matchings of size  $L$ . We'll first show that:
Lemma 4 (Existence of an optimal matching) If  $\mathcal{X} = [0,1]^d$  for some  $d\geq 1$
$$
d _ {\frac {n}{4}} ^ {*} = O \left(n ^ {- \frac {1}{d}}\right) \tag {28}
$$
with probability 1.
That is, there exists a matching of size at least  $\frac{n}{4}$  such that the maximum pairwise distance in this matching scales like  $O\left(n^{-\frac{1}{d}}\right)$ . Lemma 4 demonstrates the existence of a sizable matching in which the maximum pairwise distance indeed tends to 0. We next demonstrate that this approximates the optimal matching, at the cost of a factor of 2 on  $L$ .
Lemma 5 (Greedy approximation to the optimal matching)
$$
\max  _ {l \in [ L ]} \| x _ {2 l - 1} - x _ {2 l} \| _ {2} \leq d _ {2 L} ^ {*} \tag {29}
$$
That is, the maximum distance between any of the  $L$  pairs of observations chosen by our algorithm will be no more than the maximum such distance in the optimal matching of size  $2L$ .
Corollary 5.1 For  $L \leq \frac{n}{8}$ , we have:
$$
\max  _ {l \in [ L ]} \| x _ {2 l - 1} - x _ {2 l} \| _ {2} = O \left(n ^ {- \frac {1}{d}}\right) \tag {30}
$$
This follows immediately by invoking lemma 4 to bound the right hand side of lemma 5. Corollary 5.1 demonstrates that as  $n$  grows large, the maximum pairwise  $\ell_2$  distance between  $L$  greedily chosen pairs will go to zero at a rate of  $O\left(n^{-\frac{1}{d}}\right)$  provided  $L \leq \frac{n}{8}$ . We now show that the smoothness condition (12) further implies that, under these same conditions, we recover the asymptotic validity guarantee (13).
From approximately optimal pairings to asymptotic validity.
With the previous lemmas in place, the proof of theorem 2 is straightforward. Plugging the smoothness condition (12) into the definition of the odds ratio (9) yields the following:
For all  $(x_{2\ell -1},y_{2\ell -1}),(x_{2\ell},y_{2\ell})$
$$
r \left(\left(x _ {2 \ell - 1}, y _ {2 \ell - 1}\right), \left(x _ {2 \ell}, y _ {2 \ell}\right)\right) \in \left[ \frac {1}{\left(1 + C \left| | x _ {2 \ell - 1} - x _ {2 \ell} \right| _ {2}\right) ^ {2}}, \left(1 + C \left| | x _ {2 \ell - 1} - x _ {2 \ell} \right| _ {2}\right) ^ {2} \right] \tag {31}
$$
Where  $C > 0$  is the same constant in the definition of the smoothness condition (12). Corollary 5.1 shows that  $\| x_{2\ell -1} - x_{2\ell}\| _2 = O\left(n^{-\frac{1}{d}}\right)$ , so (31) immediately implies that  $\varepsilon_{n,L}^{*}$ , defined in (11), also goes to zero at a rate of  $O\left(n^{-\frac{1}{d}}\right)$ . Thus, if we take  $L$  to be a constant and  $K\to \infty$ , the type I error given in (10) can be rewritten as
$$
\begin{array}{l} \mathbb {P} \left(\tau_ {K} \leq \alpha\right) \leq \alpha + \left(1 - \left(1 - \varepsilon_ {n, L} ^ {*}\right) ^ {L}\right) + \frac {1}{K + 1} (32) \\ \leq \alpha + \varepsilon_ {n, L} ^ {*} L + \frac {1}{K + 1} (33) \\ = \alpha + O \left(n ^ {- \frac {1}{d}}\right) (34) \\ \end{array}
$$
In principle, we could find this optimal matching by binary searching for  $d_{L}^{*}$  using the non-bipartite maximal matching algorithm of Edmonds (1965); for simplicity, our implementation uses a greedy matching strategy instead.
Where (33) follows from corollary 3.1. If we instead allow  $L$  to scale like  $o(n^{\frac{1}{d}})$  (still taking  $K \to \infty$ ), (33) implies:
$$
\mathbb {P} \left(\tau_ {K} \leq \alpha\right) \leq \alpha + o (1) \tag {35}
$$
which concludes the proof of theorem 2.
D Proofs of auxiliary lemmas
Recall that one definition of the total variation distance between two distributions  $P$  and  $Q$  is to consider the set of couplings on these distributions. In particular, the total variation distance can be equivalently defined as:
$$
\operatorname {T V} (P, Q) = \inf  _ {(X, Y) \sim C (P, Q)} \mathbb {P} (X \neq Y) \tag {36}
$$
Where  $C(\cdot, \cdot)$  is the set of couplings on its arguments. Consider then the following straightforward coupling on  $X \coloneqq (u_1 \ldots u_L)$  and  $Y \coloneqq (v_1 \ldots v_L)$ : draw  $L$  random numbers independently and uniformly from the interval [0, 1]. Denote these by  $c_1 \ldots c_L$ . Let  $u_i = \mathbb{1}[c_i \leq p_i]$ , and  $v_i = \mathbb{1}[c_i \leq q_i]$ . It's clear that  $X$  and  $Y$  are marginally distributed according to  $p_1 \ldots p_L$  and  $q_1 \ldots q_L$ , respectively. Furthermore, the probability that  $u_i \neq v_i$  is  $|p_i - q_i|$  by construction. Thus we have:
$$
\mathbb {P} (X \neq Y) = 1 - \mathbb {P} (X = Y) = 1 - \Pi_ {i \in [ L ]} \left(1 - \left| p _ {i} - q _ {i} \right|\right) \leq 1 - \left(1 - \max  _ {i} \left| p _ {i} - q _ {i} \right|\right) ^ {L} \tag {37}
$$
Proof of Corollary 3.1.
In the preceding proof of lemma 3, observe that we could have instead written:
$$
\mathbb {P} (X \neq Y) = \bigcup_ {i \in [ L ]} \left\{v _ {i} \neq u _ {i} \right\} \underbrace {\leq} _ {\text {u n i o n b o u n d}} \sum_ {i \in [ L ]} | p _ {i} - q _ {i} | \leq L \max  _ {i \in [ L ]} | p _ {i} - q _ {i} | \tag {38}
$$