SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 18. [67] We have  $E[h_{\langle i \rangle}] = p_*$  for  $i \in [s]$  and  $h_{\langle 1 \rangle}, h_{\langle 2 \rangle}, \dots, h_{\langle s \rangle}$  are mutually independent, if  $\pmb{\mu}_{\hat{\mathbb{P}}_{\mathcal{B}_{\langle i \rangle}}} = \pmb{\mu}_{\hat{\mathbb{Q}}_{\mathcal{B}_{\langle i \rangle}}}$  for every  $i \in [s]$  and the  $p$ -values are uniformly distributed.
Denote by  $\widetilde{\mathcal{B}}$  the set of rectangle regions that the local two samples  $\hat{X}_{\mathcal{B}_{\langle i\rangle}}^{\prime}$  and  $\hat{Y}_{\mathcal{B}_{\langle i\rangle}}^{\prime}$  are actually drawn from one identical distribution, and we define
$$
\mathcal {H} _ {0} = \left\{H _ {0, \langle i \rangle}: \mathcal {B} _ {\langle i \rangle} \in \widetilde {\mathcal {B}} \right\}.
$$
Lemma 19. [67] If there is some rectangle region such that  $\pmb{\mu}_{\hat{\mathbb{P}}_{\mathcal{B}_{\langle \cdot \rangle}}} \neq \pmb{\mu}_{\hat{\mathbb{Q}}_{\mathcal{B}_{\langle \cdot \rangle}}}$ , and if the  $p$ -value follows a uniform distribution under null hypothesis  $H_{0,\langle \cdot \rangle}$ , then we have
$$
E \left[ h _ {\langle i \rangle} \mid \left\{h _ {\langle k \rangle} \right\} _ {k = i + 1} ^ {s}, \left\{\mathbb {I} \left(H _ {0, \langle k \rangle} \in \mathcal {H} _ {0}\right) \right\} _ {k = i + 1} ^ {s}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0} \right] = p _ {*} \quad f o r \quad i \in [ s ].
$$
We also define the weighted mirror-conservativeness, motivated from [85, 67], as follows.
Definition 20. We say that a density function  $f(\cdot)$  satisfies the weighted mirror-conservativeness if it holds that, for some given  $p \in (0,1]$ ,
$$
f (a w) \leq f (1 - (1 - w p) a / p) \text {f o r e v e r y} w \in [ 1, 2 ], a \in [ 0, p ].
$$
Here, we introduce an additional parameter  $w$  to incorporate two directions for our method, which is different from the previous mirror-conservativeness [85, 67]. We could also present two sufficient conditions for weighted mirror-conservativeness from [85, 67]: i) the non-decrease of  $f$  and ii) the convexity of cumulative density function of  $p$ -value.
Lemma 21. If the density function of the  $p$ -value satisfies the weighted mirror-conservativeness under local null hypothesis  $H_{0,\langle \cdot \rangle}$ , and if there is some rectangle region such that  $\mu_{\hat{\mathbb{P}}_{B_{\langle \cdot \rangle}}} \neq \mu_{\hat{\mathbb{Q}}_{B_{\langle \cdot \rangle}}}$  then we have, for every  $i \in [s]$
$$
E \left[ h _ {\langle i \rangle} \mid \left\{h _ {\langle k \rangle} \right\} _ {k = i + 1} ^ {s}, \left\{\mathbb {I} \left(h _ {\langle k \rangle} \in \mathcal {H} _ {0}\right) \right\} _ {k = i + 1} ^ {s}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \left\{g _ {\langle k \rangle} \right\} _ {k = 1} ^ {s} \right] \leq p _ {*}, \tag {27}
$$
where  $g_{\langle k\rangle} = g(\hat{X}_{\mathcal{B}_{\langle k\rangle}}^{\prime},\hat{Y}_{\mathcal{B}_{\langle k\rangle}}^{\prime})$
Proof. We first prove
$$
E \left[ h _ {\langle i \rangle} \mid g _ {\langle i \rangle} = a \right] \leq p _ {*} \quad \text {f o r} \quad H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \tag {28}
$$
from our bi-directional hypothesis and masked  $p$ -value with  $\beta \in [1,2]$  in Eqn. (7), and it is sufficient to consider the following two cases:
For  $F_{\mathcal{B}_{\langle i\rangle}}^{\top}L_{\mathcal{B}_{\langle i\rangle}}^{\prime -1}(c_{\hat{X}_{\mathcal{B}_{\langle i\rangle}}^{\prime}} - c_{\hat{Y}_{\mathcal{B}_{\langle i\rangle}}^{\prime}})\geq 0$  we have
$$
\begin{array}{l} E \left[ h _ {\langle i \rangle} \mid g _ {\langle i \rangle} = a, \boldsymbol {F} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\top} \boldsymbol {L} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}}\right) > 0 \right] \\ = \frac {p _ {*} f (\beta a)}{p _ {*} f (\beta a) + (1 - p _ {*}) f (1 - \frac {1 - \beta p _ {*}}{p _ {*}} a)} \\ = \frac {p _ {*}}{p _ {*} + (1 - p _ {*}) f \left(1 - \frac {1 - \beta p _ {*}}{p _ {*}} a\right) / f (\beta a)} \\ \leq \quad p _ {*}, \\ \end{array}
$$
where the last inequality holds from the weighted mirror-conservativeness with  $p$ -value's uniform distribution  $\mathcal{U}[0,1]$ .
For  $\pmb{F}_{\mathcal{B}_{\langle i\rangle}}^{\top}\pmb{L}_{\mathcal{B}_{\langle i\rangle}}^{\prime -1}(\pmb{c}_{\hat{X}_{\mathcal{B}_{\langle i\rangle}}^{\prime}} - \pmb{c}_{\hat{Y}_{\mathcal{B}_{\langle i\rangle}}^{\prime}}) < 0$  we similarly have
$$
\begin{array}{l} E \left[ h _ {\langle i \rangle} \mid g _ {\langle i \rangle} = a, \boldsymbol {F} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\top} \boldsymbol {L} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}}\right) <   0 \right] \\ = \frac {p _ {*} f ((2 - \beta) a)}{p _ {*} f ((2 - \beta) a) + (1 - p _ {*}) f (1 - \frac {1 - (2 - \beta) p _ {*}}{p _ {*}} a)} \\ = \frac {p _ {*}}{p _ {*} + (1 - p _ {*}) f \left(1 - \frac {1 - (2 - \beta) p _ {*}}{p _ {*}} a\right) / f ((2 - \beta) a)} \\ \leq \quad p _ {*}. \\ \end{array}
$$
We define the information available for choosing  $H_{0,\langle i\rangle}$  as a filtration (sequence of nested  $\sigma$ -fields)
$$
\mathcal {F} _ {\langle i \rangle} = \sigma \left(\{\mathcal {B} _ {\langle k \rangle}, g _ {\langle k \rangle} \} _ {k = 1} ^ {s}, \{\chi_ {\ell} ^ {2} (\mathcal {T} (\hat {X} _ {\mathcal {B} _ {\langle k \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle k \rangle}} ^ {\prime})) \} _ {k = i + 1} ^ {s}\right),
$$
and also define the filtration
$$
\mathcal {F} _ {\langle i \rangle} ^ {h} = \sigma \left(\left\{h _ {\langle k \rangle} \right\} _ {k = i + 1} ^ {s}, \left\{\mathbb {I} \left(H _ {0, \langle k \rangle} \in \mathcal {H} _ {0}\right) \right\} _ {k = i + 1} ^ {s}\right).
$$
$$
\begin{array}{l} E \left[ h _ {\langle i \rangle} \mid \left\{h _ {\langle k \rangle} \right\} _ {k = i + 1} ^ {s}, \left\{\mathbb {I} \left(h _ {\langle k \rangle} \in \mathcal {H} _ {0}\right) \right\} _ {k = i + 1} ^ {s}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \left\{g _ {\langle k \rangle} \right\} _ {k = 1} ^ {s} \right] \\ = E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ = E \left[ E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \mid \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ \end{array}
$$
where the last equality holds from the law of total expectation. We have
$$
\begin{array}{l} E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ = \sum_ {H _ {0, j} \in \mathcal {R} (i) \cap \mathcal {H} _ {0}} E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \left\{g _ {\langle k \rangle} \right\} _ {k = 1} ^ {s} \right] \\ \times \Pr \left[ H _ {0, \langle i \rangle} = H _ {0, j} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ = \sum_ {H _ {0, j} \in \mathcal {R} (i) \cap \mathcal {H} _ {0}} E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle} \right] \Pr \left[ H _ {0, \langle i \rangle} = H _ {0, j} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right], \\ \end{array}
$$
where the last equation holds from the fact that  $\{\mathcal{F}_{\langle i\rangle}^h,H_{0,\langle i\rangle}\in \mathcal{H}_0,\{g_{\langle k\rangle}\}_{k = 1}^s\}$  is a subset of  $\mathcal{F}_{\langle i\rangle}$ . We further have, since  $h_{\langle i\rangle}$  is independent of other information in  $\mathcal{F}_{\langle i\rangle}$ ,
$$
\begin{array}{l} E \left[ h _ {\langle i \rangle} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ = \sum_ {H _ {0, j} \in \mathcal {R} (i) \cap \mathcal {H} _ {0}} E \left[ h _ {\langle i \rangle} \mid g _ {\langle i \rangle} \right] \Pr \left[ H _ {0, \langle i \rangle} = H _ {0, j} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ \leq \quad p _ {*} \sum_ {H _ {0, j} \in \mathcal {R} (i) \cap \mathcal {H} _ {0}} \Pr \left[ H _ {0, \langle i \rangle} = H _ {0, j} \mid \mathcal {F} _ {\langle i \rangle}, \mathcal {F} _ {\langle i \rangle} ^ {h}, H _ {0, \langle i \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \\ = \quad p _ {*}, \\ \end{array}
$$
We say that a random variable  $Z$  follows a Bernoulli distribution with parameter  $p$ , denoted by  $Z \sim \mathcal{B}ern(p)$ , if
$$
\Pr [ Z = 1 ] = p \quad \text {a n d} \quad \Pr [ Z = 0 ] = 1 - p.
$$
We also say that a random variable  $Z$  follows a negative binomial distribution with parameters  $r$  and  $p$ , denoted by  $Z \sim \mathcal{NB}(r, p)$ , if
$$
\operatorname * {P r} [ Z = k ] = \binom {k + r - 1} {k} (1 - p) ^ {r} p ^ {k}  .
$$
It is necessary to introduce a definition as follows:
Definition 22. We say that a random variable  $Z$  is stochastically dominated by a distribution  $\mathbb{G}$ , denoted by
$$
Z \preceq \mathbb {G},
$$
if for random variable  $X\sim \mathbb{G}$  , it holds that
$$
\Pr [ Z \geq x ] \leq \Pr [ X \geq x ] \quad \text {f o r} \quad x \in (- \infty , + \infty).
$$
We further introduce some useful lemmas as follows:
Lemma 23. [67] Let  $Z_{1}, \dots, Z_{s}$  be i.i.d random variables with  $Z_{i} \sim \mathcal{B}\text{ern}(p_{*})$  for some  $p_{*} > 0$ , and write  $\widetilde{N}_{t} = \sum_{j=1}^{t} Z_{j}$  and  $\widetilde{\mathcal{G}}_{t} = \sigma(\widetilde{N}_{t}, \{Z_{j}\}_{j=t+1}^{s})$  for  $t \in [s]$ . We have
$$
\widetilde {N} _ {\tilde {\tau}} \preceq \mathcal {N B} (v, p _ {*}),
$$
where the stopping index  $\widetilde{\tau}$  is parameterized by some constant  $v(\geq 1)$ , defined by
$$
\widetilde {\tau} = \max  \left\{0 <   t \leq s: t - \widetilde {N} _ {t} <   v o r t = 1 \right\}.
$$
We further introduce a weighted version of Lemma 23 as follows:
Lemma 24. [67] Let  $\{W_j\}_{j=1}^s$  be a sequence of weights, drawn from a Bernoulli distribution, s.t.  $\sum_{j=1}^s W_j = u$  for fixed constant  $u \leq s$ ; and  $Z_j \mid \sigma(\{Z_k, W_k\}_{k=j+1}^s, W_j = 1) \sim \mathcal{B}ern(p_*)$ . Write  $N_t^w = \sum_{j=1}^t W_j Z_j$ , and we have
$$
N _ {\tau^ {w}} ^ {w} \preceq \mathcal {N B} (v, p _ {*}),
$$
where the stopping index  $\tau^w$  is parameterized by some constant  $v(\geq 1)$ , defined by
$$
\tau^ {w} = \max  \left\{0 <   t \leq s: \sum_ {j = 1} ^ {t} W _ {j} - N _ {t} ^ {w} <   v o r t = 1 \right\}.
$$
We now introduce a different version of Lemma 24 by considering different parameters for Bernoulli distribution as follows:
Lemma 25. [67] Let  $Z_{j} \mid \sigma\left(\{Z_{k}, W_{k}\}_{k=j+1}^{s}, W_{j}=1\right)$  follow a Bernoulli distribution with parameter  $p\left(\{Z_{k}, W_{k}\}_{k=j+1}^{s}\right)$  for  $j \in [s]$ , respectively. We have
$$
N _ {\tau^ {w}} ^ {w} \preceq \mathcal {N B} \left(v, p \left(\left\{Z _ {k}, W _ {k} \right\} _ {k = j + 1} ^ {s}\right)\right) \preceq \mathcal {N B} (v, p _ {*}),
$$
if  $p(\{Z_k, W_k\}_{k = j + 1}^s) \leq p_*$  for every  $j \in [s]$ .
We now present the detailed proof of Theorem 5 as follows.
Proof of Theorem 5. We first consider  $\mu_{\hat{\mathbb{P}}_{B_{\langle i\rangle}}} = \mu_{\hat{\mathbb{Q}}_{B_{\langle i\rangle}}}$  for every  $i\in [s]$ . From Lemma 18,  $\{h_{\langle i\rangle}\}_{i = 1}^{s}$  are  $s$  i.i.d. random variables with  $h_{\langle i\rangle}\sim \mathcal{B}ern(p_{*})$ . Recall that the stopping rule in our testing, i.e., Eqn. (11), which is equivalent to
$$
1 - \left(1 - p _ {*}\right) ^ {t - | \mathcal {I} (t) | + 1} \leq \alpha_ {*},
$$
where  $\alpha_{*}$  is a parameter to control familywise error rate and  $\mathcal{I}(t) = \{i\in [t]\colon h(\hat{X}_{\mathcal{B}_{\langle i\rangle}}^{\prime},\hat{Y}_{\mathcal{B}_{\langle i\rangle}}^{\prime}) = 1\}$ . The stopping rule can be rewritten as  $t - |\mathcal{I}(t)| < v$  with
$$
v = \left\lfloor \ln \left(1 - \alpha_ {*}\right) / \ln \left(1 - p _ {*}\right) \right\rfloor . \tag {29}
$$
Let  $Z_{j} = h_{\langle j\rangle}$  and  $\widetilde{N}_{t} = \sum_{j = 1}^{t}Z_{j}$ . We define the stopping index  $\widetilde{\tau}$  as follows
$$
\widetilde {\tau} = \max  \left\{0 <   t \leq s: t - \widetilde {N} _ {t} <   v \text {o r} t = 1 \right\}.
$$
From Lemma 23, the number of rejections at the stopping index is given by
$$
| \mathcal {I} (\widetilde {\tau}) | = \sum_ {j = 1} ^ {\widetilde {\tau}} h _ {\langle j \rangle} = \widetilde {N _ {\widetilde {\tau}}} \preceq \mathcal {N B} (v, p _ {*}).
$$
If  $\pmb{\mu}_{\hat{\mathbb{P}}_{\mathcal{B}_{\langle i\rangle}}} = \pmb{\mu}_{\hat{\mathbb{Q}}_{\mathcal{B}_{\langle i\rangle}}}$  for every  $i\in [s]$ , then the number of false rejections is
$$
\left| \mathcal {I} (\tilde {\tau}) \cap \mathcal {H} _ {0} \right| = \left| \mathcal {I} (\tilde {\tau}) \right| \preceq \mathcal {N B} (v, p _ {*}),
$$
and hence the familywise error rate (FWER) is upper bounded by
$$
\Pr \left[ | \mathcal {I} (\widetilde {\tau}) \cap \mathcal {H} _ {0} | \geq 1 \right] \leq 1 - \left(1 - p _ {*}\right) ^ {v} \leq \alpha_ {*},
$$
where the last inequality follows from Eqn. (29).
We now consider that there is some rectangle region with  $\mu_{\hat{\mathbb{P}}_{\mathcal{B}_{\langle \cdot \rangle}}} \neq \mu_{\hat{\mathbb{Q}}_{\mathcal{B}_{\langle \cdot \rangle}}}$ . In such case, we provide an upper bound for familywise error rate without the information of masked  $p$ -values, and prove that the number of false rejections is stochastically dominated by  $\mathcal{NB}(v,p_{*})$ .
Let  $Z_{j} = h_{\langle j\rangle}$  and  $W_{j} = \mathbb{I}(H_{0,\langle j\rangle}\in \mathcal{H}_{0})$  . We define the stopping index  $\tau^w$  as follows:
$$
\tau^ {w} = \max  \left\{0 <   t \leq s: \sum_ {j = 1} ^ {t} \mathbb {I} [ h _ {\langle j \rangle} = 0 \cap H _ {0, \langle j \rangle} \in \mathcal {H} _ {0} ] = \sum_ {j = 1} ^ {t} W _ {j} (1 - Z _ {j}) <   v \text {o r} t = 1 \right\},
$$
where  $v$  is given in Eqn. (29). It is easy to see that
$$
Z _ {j} \mid \sigma \left(\left\{Z _ {k}, W _ {k} \right\} _ {k = j + 1} ^ {s}, W _ {j} = 1\right) \sim \mathcal {B e r n} (p _ {*})
$$
from Lemma 19. Denote by  $u = |\mathcal{H}_0|$ , and we have  $\sum_{j=1}^{s} W_j = u$  and  $u \leq s$ . From Lemma 24, we have the number of false rejections
$$
\sum_ {j = 1} ^ {\tau^ {w}} \mathbb {I} \left[ h _ {\langle j \rangle} = 1 \cap H _ {0, \langle j \rangle} \in \mathcal {H} _ {0} \right] = \sum_ {j = 1} ^ {\tau^ {w}} W _ {j} Z _ {j} = N _ {\tau^ {w}} ^ {w} \preceq \mathcal {N B} (v, p _ {*}). \tag {30}
$$
Recall that  $t - |\mathcal{I}(t)| < v$  is our stopping rule on the exploration of local significant differences. Denote by  $\tau_T^w$  the stopping index in our exploration, and we have
$$
\begin{array}{l} \sum_ {j = 1} ^ {\tau_ {T} ^ {w}} \mathbb {I} [ h _ {\langle j \rangle} = 0 \cap H _ {0, \langle j \rangle} \in \mathcal {H} _ {0} ] \\ \leq \sum_ {j = 1} ^ {\tau_ {T} ^ {w}} \mathbb {I} [ h _ {\langle j \rangle} = 0 ] = \tau_ {T} ^ {w} - \sum_ {j = 1} ^ {\tau_ {T} ^ {w}} \mathbb {I} [ h _ {\langle j \rangle} = 1 ] = \tau_ {T} ^ {w} - \mathcal {I} (\tau_ {T} ^ {w}) <   v. \\ \end{array}
$$
Since  $N_{t}^{w}$  is non-decreasing with respect to  $t$ , it is easy to obtain
$$
\tau_ {T} ^ {w} \leq \tau^ {w} \quad \text {a n d} \quad N _ {\tau_ {T} ^ {w}} ^ {w} \leq N _ {\tau^ {w}} ^ {w},
$$
and we have the number of false rejections
$$
\left| \mathcal {I} \left(\tau_ {T} ^ {w}\right) \cap \mathcal {H} _ {0} \right| = \sum_ {j = 1} ^ {\tau_ {T} ^ {w}} \mathbb {I} \left[ h _ {\langle j \rangle} = 1 \cap H _ {0, \langle j \rangle} \in \mathcal {H} _ {0} \right] = N _ {\tau_ {T} ^ {w}} ^ {w} \leq N _ {\tau^ {w}} ^ {w} \preceq \mathcal {N B} (v, p _ {*}). \tag {31}
$$
We upper bound the familywise error rate without considering the masked  $p$ -values as follows:
$$
\Pr \left[ | \mathcal {I} \left(\tau_ {T} ^ {w}\right) \cap \mathcal {H} _ {0} | \geq 1 \right] \leq \Pr \left[ | \mathcal {I} \left(\tau^ {w}\right) \cap \mathcal {H} _ {0} | \geq 1 \right] \leq 1 - (1 - p _ {*}) ^ {v} \leq \alpha_ {*}.
$$
We finally take masked  $p$ -values  $\{g_{\langle k\rangle}\}_{k = 1}^{s}$  into consideration. From Lemma 21, it is easy to observe
$$
Z _ {j} \mid \sigma \big (\{Z _ {k}, W _ {k} \} _ {k = j + 1} ^ {s}, W _ {j} = 1 \big) \sim \mathcal {B e r n} \big (p \big (\{Z _ {k}, W _ {k} \} _ {k = j + 1} ^ {s} \big) \big),
$$
$$
p (\{Z _ {k}, W _ {k} \} _ {k = j + 1} ^ {s}) = E \left[ h _ {\langle j \rangle} \mid \{h _ {\langle k \rangle} \} _ {k = j + 1} ^ {s}, \{\mathbb {I} (h _ {\langle k \rangle} \in \mathcal {H} _ {0}) \} _ {k = j + 1} ^ {s}, H _ {0, \langle j \rangle} \in \mathcal {H} _ {0}, \{g _ {\langle k \rangle} \} _ {k = 1} ^ {s} \right] \leq p _ {*}.
$$
$$
\mathcal {N B} \left(v, p \left(\left\{Z _ {k}, W _ {k} \right\} _ {k = j + 1} ^ {s}\right)\right) \preceq \mathcal {N B} (v, p _ {*}),
$$
and we further have, from Lemma 25,
$$
\left| \mathcal {I} \left(\tau_ {T} ^ {w}\right) \cap \mathcal {H} _ {0} \right| = \sum_ {j = 1} ^ {\tau^ {w}} W _ {j} Z _ {j} = N _ {\tau^ {w}} ^ {w} \preceq \mathcal {N B} \left(v, p \left(\left\{Z _ {k}, W _ {k} \right\} _ {k = j + 1} ^ {s}\right)\right) \preceq \mathcal {N B} (v, p _ {*}).
$$
We finally upper bound the familywise error rate by considering the masked  $p$ -values  $\left\{g_{\langle k\rangle}\right\}_{k = 1}^{s}$  as
$$
\begin{array}{l} \Pr \left[ | \mathcal {I} \left(\tau_ {T} ^ {w}\right) \cap \mathcal {H} _ {0} | \geq 1 \mid \left\{g _ {\langle k \rangle} \right\} _ {k = 1} ^ {s} \right] \leq \mathbb {P} \left(| \mathcal {I} (\tau^ {w}) \cap \mathcal {H} _ {0} | \geq 1 \mid \left\{g _ {\langle k \rangle} \right\} _ {k = 1} ^ {s}\right) \\ \leq 1 - \left(1 - p _ {*}\right) ^ {v} \leq \alpha_ {*}. \\ \end{array}
$$
B Optimization for Test Locations and Mahalanobis Kernels
We take gradient method [43] for the optimization of Eqn. (5) as in the work of [10]. Specifically, we calculate gradients, and update test locations and Mahalanobis kernels iteratively. In the following of this section, we present the calculation of some crucial gradients in optimization.
For test location  $\pmb{v}_j$  with  $j\in [\ell ]$ , we have
$$
\nabla_ {\boldsymbol {v} _ {j}} \mathcal {T} (\hat {X}, \hat {Y}) = \left(\frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {v} _ {j , 1}}, \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {v} _ {j , 2}}, \dots , \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {v} _ {j , \ell}}\right) ^ {\top}, \tag {32}
$$
where, for  $i\in [\ell ]$
$$
\frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {v} _ {j , i}} = \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {X}}} \frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial \boldsymbol {v} _ {j , i}} + \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {Y}}} \frac {\partial \boldsymbol {c} _ {\hat {Y}}}{\partial \boldsymbol {v} _ {j , i}} + \operatorname {T r} \left[ \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \Sigma_ {\hat {X} , \hat {Y}}} \frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial \boldsymbol {v} _ {j , i}} \right], \tag {33}
$$
where  $\mathrm{Tr}[\cdot ]$  denotes the trace.
$$
\begin{array}{l} \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {X}}} = 2 m n \Sigma_ {\hat {X}, \hat {Y}} ^ {- 1} (\boldsymbol {c} _ {\hat {X}} - \boldsymbol {c} _ {\hat {Y}}) / (m + n), \\ \frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial \boldsymbol {v} _ {j , i}} = \frac {1}{m} \sum_ {r = 1} ^ {m} \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial \boldsymbol {v} _ {j , i}} \quad \text {w i t h} \quad \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial \boldsymbol {v} _ {j , i}} = \left(0, \dots , 0, \frac {\partial \kappa_ {j} (\hat {\boldsymbol {x}} _ {r} , \boldsymbol {v} _ {j})}{\partial \boldsymbol {v} _ {j , i}}, 0, \dots , 0\right) ^ {\top}, \\ \end{array}
$$
where all elements are zeros except for the  $j$ -th element. We also have
$$
\begin{array}{l} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial \boldsymbol {v} _ {j , i}} = \frac {\partial}{\partial \boldsymbol {v} _ {j , i}} \left\{\exp \left(- \left(\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}\right) ^ {\top} M _ {j} \left(\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}\right) / 2 \gamma_ {j} ^ {2}\right) \right\} \\ { = } { \kappa _ { j } ( \hat { \boldsymbol { x } } _ { r } , \boldsymbol { v } _ { j } ) ( M _ { j } ( \hat { \boldsymbol { x } } _ { r } - \boldsymbol { v } _ { j } ) ) ^ { \top } ( 0 , \ldots , 1 , \ldots , 0 ) ^ { \top } / \gamma _ { j } ^ { 2 } , } \\ \end{array}
$$
where the  $i$ -th element is 1. We similarly calculate  $\partial \mathcal{T}(\hat{X},\hat{Y}) / \partial c_{\hat{Y}}\times \partial c_{\hat{Y}} / \partial v_{j,i}$ .
For the third term in Eqn. (33), we have
$$
\begin{array}{l} \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \Sigma_ {\hat {X} , \hat {Y}}} = \Sigma_ {\hat {X}, \hat {Y}} ^ {- 1} (\boldsymbol {c} _ {\hat {X}} - \boldsymbol {c} _ {\hat {Y}}) (\boldsymbol {c} _ {\hat {X}} - \boldsymbol {c} _ {\hat {Y}}) ^ {\top} \Sigma_ {\hat {X}, \hat {Y}} ^ {- 1}, \\ \frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial \boldsymbol {v} _ {j , i}} = \frac {1}{m + n - 2} \left(\sum_ {r = 1} ^ {m} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} + \sum_ {r = 1} ^ {n} \frac {\partial (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) ^ {\top}}{\partial \boldsymbol {v} _ {j , i}}\right), \\ \end{array}
$$