SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use a standard deep Conv-net [40], which has been employed for SOTA GAN discriminator tasks in similar settings. It has four convolutional layers and one fully connected layer outputting the feature space of size  $(300,1)$ . For SCHE and LBI, we simply added a linear layer of  $(300,2)$  after applying ReLU to the 300-dimensional layer and used the cross-entropy loss to train the network. Note that this is equivalent to first fixing the feature space and then performing logistic regression to the feature space. For kernels, we add extra trainable parameters after the 300-d feature output.
For the MMD-based tests, we simply train the kernel on the neural net and evaluate our objective. For UME, we used a slightly generalized version of the original statistic in [31] which allows for comparison on randomly selected witness locations in the null hypothesis with  $m \neq n$  (see Appendix F.5). The kernel is trained using our heuristic (see (8) and Appendix F.2), with MMD replaced by UME. The formula for UME variance can be found in [31]. For RFM, we use Algorithm 1 in [44] to learn a kernel on (stochastic batched) samples, and then use our MMD test on the trained kernel.
We use 80 training epochs for most of our code from the CNN architecture (for classifiers, this is well after interpolating the training data and roughly when validation loss stops decreasing), and a batch size of 32 which has a slight empirical benefit compared to larger batch sizes. The learning rates are tuned separately in MMD methods for optimality, whereas for classifiers they follow the discriminator's original setting from [40]. In Phase 2 of Algorithm 1, we choose  $k = 1000$  for the
Figure 6: Relevant plots following the setting in Figure 3 (in the main text) of fixing  $n_{\mathrm{tr}} = 1920$  and varying sample size  $m$  in the x-axis for the comparison with missing benchmarks. Errorbars are projected showing standard deviation across 10 runs. We replaced part (d) in Figure 3 (in the main text) to a sanity check in our FPR when thresholded at  $\alpha = 0.05$ .
desired precision while not compromising runtime. For each task, we run 10 independent models and report their performances as the mean and standard deviation of those 10 runs as estimates. We refer to a full set of hyper-parameters in our code implementation.
Our code is implemented in Python 3.7 (PyTorch 1.1) and was ran on an NVIDIA RTX 3080 GPU equipped with a standard torch library and dataset extensions. Our code setup for feature extraction is similar to that of [39]. For benchmark implementations, our code follows from the original code templated provided by the cited papers.
G.3 Sample Allocation
We make a comment on why (4) is different from just thresholding  $\widehat{\mathrm{MMD}^2}(Z, Y^{\mathrm{tr}}) - \widehat{\mathrm{MMD}^2}(Z, X^{\mathrm{tr}})$  at 0, which was what we did in part (c) of Figure 3 (and hence the difference along the curve of MMD-M vs Figure 1.2). Our theory assumes that the samples are i.i.d. conditioned on the kernel being chosen already. However, in the experiments, the kernel is dependent on the training data. Therefore, to evaluate the MMD estimate (between experimentations), one needs extra data that does not intersect with training.
In fact, it can be experimentally shown by comparing Figure 1.2 and Figure 2(c) that doing so (while reducing the sample complexity on  $n_{\mathrm{ev}}$ ) hurts performance. Indeed, we found out that when  $X^{\mathrm{ev}}, Y^{\mathrm{ev}}$  are non-intersecting with training, performance is (almost) always better at a cost of hurting the overall sample complexity of  $n$ .
G.4 Remarks on Results
Figure 6 lists all of our benchmarks in the setting of Figure 3 (in the main text) on missing benchmarks, where the last figure is replaced by the false positive rate at thresholding at  $\alpha = 0.05$  to verify our results. As mentioned in the main text, our MMD-M method consistently outperforms other benchmarks on both the expected  $p$ -value (of alternate) and rejection rate at  $\alpha = 0.05$ , while all of our tests observe a empirical false positive rate close to  $\alpha = 0.05\%$  (Part (b)), showing the consistency of methods.
H Application: Higgs-Boson Detection
H.1 Dataset Details
We use the Higgs dataset available online at http://archive.ics.uci.edu/ml/datasets/HIGGS, produced using Monte Carlo simulations [5]. The dataset is nearly balanced, containing 5,829,122 signal instances and 5,170,877 background instances. Each instance is a 28-dimensional vector, consisting of 28 features. The first 21 features are kinematic properties measured by the detectors in the accelerator, such as momentum and energy. The last 7 properties are invariant masses, derived from the first 21 features.
Figure 7: This figure visualizes the distribution of the 26th feature, the invariant mass  $m_{Wbb}$ . The red and black lines are the histograms of the original dataset. We employ MMD-M as a classifier, trained and evaluated using  $n_{\mathrm{tr}} = 1.3 \times 10^6$  and  $n_{\mathrm{ev}} = n_{\mathrm{opt}} = 2 \times 10^4$  through Algorithm 3. The blue(green) line represents all instances  $z$ 's whose "witness scores"  $f(z; X^{\mathrm{ev}}, Y^{\mathrm{ev}})$ 's are larger(smaller) than  $t_{\mathrm{opt}}$ .
H.2 Experiment Setup and Training Models
The modified Algorithm 1 is shown in Algorithm 2 and Algorithm 3. Compared with Algorithm 2, we implement the thresholding trick (Section 4.3) in Algorithm 3.
H.2.1 Configuration and Model Architecture
We implement all methods in Python 3.9 and PyTorch 1.13 and run them on an NVIDIA Quadro RTX 8000 GPU.
For all classifier-based methods in this study (SCHE and LBI), we adopt the same architecture as previously proposed in [5]. The classifiers are six-layer neural networks with 300 hidden units in each layer, all employing the tanh activation function. For SCHE, the output layer is a single sigmoid unit and we utilize the binary cross-entropy loss for training. For LBI, the output layer is a linear unit and we utilize the binary cross entropy loss combined with a logit function (which is more numerically stable than simply using a sigmoid layer followed by a cross entropy loss).
For all MMD-based methods (MMD-M, MMD-G, MMD-O, and UME), the networks  $\varphi$  and  $\varphi'$  are both six-layer neural networks with 300 ReLU units in each layer. The feature space, which is the output of the neural network  $\varphi$ , is set to be 100-dimensional. Here UME has the same kernel architecture as MMD-M, and the number of test locations is set to be  $J_{q} = 4096$ . For RFM, we adopt the same architecture as in [44], where the kernel is  $K_{M}(x,y) = \exp(-\gamma(x-y)^{T}M(x-y))$  with a constant  $\gamma$  and a learnable positive semi-definite matrix  $M$ . We set  $\gamma \equiv 1$ .
The neural networks are initialized using the default setting in PyTorch, and the bandwidths  $\sigma, \sigma'$  are initialized using the median heuristic [22]. The parameter  $\tau$  is initially set to 0.5. For UME, the witness locations  $W$  are initially randomly sampled from the training set. For RFM, the initial  $M$  equals the median bandwidth times an identity matrix.
H.2.2 Training
The size of our training set, denoted as  $n_{\mathrm{tr}}$ , varies from  $1.0 \times 10^{2}$  to  $1.6 \times 10^{6}$ . For a given  $n_{\mathrm{tr}}$ , we select the first  $n_{\mathrm{tr}}$  datapoints from each class of the Higgs dataset to form  $X^{\mathrm{tr}}$  and  $Y^{\mathrm{tr}}$ , i.e.,  $|X^{\mathrm{tr}}| = |Y^{\mathrm{tr}}| = n_{\mathrm{tr}}$ . Subsequently, we randomly select  $n_{\mathrm{validation}} = \min(\sqrt{10n_{\mathrm{tr}}}, 0.1n_{\mathrm{tr}})$  points
from each of  $X_{\mathrm{tr}}$ ,  $Y_{\mathrm{tr}}$  to constitute the validation set, while the remainder of  $X_{\mathrm{tr}}$ ,  $Y_{\mathrm{tr}}$  are used for running gradient descent. The optimizer is set to be a minibatch SGD, with a batch size of 1024, a learning rate of 0.001, and a momentum of 0.99. Training is halted once the validation loss stops to decrease for 10 epochs, then we choose the checkpoint (saved for each epoch) with the smallest validation loss thus far as our trained model. Beyond the general setting above, in RFM a batch size of 1024 doesn't work well and instead we use a batch size of 20,000.
H.3 Evaluating the Performance
H.3.1 Evaluating the p-Value with the Methodology of Algorithm 1
We call the "witness score" of an instance  $z \in \mathcal{X}$  as
$$
f (z; X ^ {\mathrm {e v}}, Y ^ {\mathrm {e v}}) = \frac {1}{n _ {\mathrm {c a l}}} \sum_ {i = 1} ^ {n _ {\mathrm {c a l}}} \left(k \left(z, Y _ {i} ^ {\mathrm {e v}}\right) - k \left(z, X _ {i} ^ {\mathrm {e v}}\right)\right). \tag {33}
$$
For a vector of instances  $Z = (Z_{1},\ldots ,Z_{m})$ , we write
$$
f (Z; X ^ {\mathrm {e v}}, Y ^ {\mathrm {e v}}) = \left(f \left(Z _ {1}; X ^ {\mathrm {e v}}, Y ^ {\mathrm {e v}}\right), \dots , f \left(Z _ {m}; X ^ {\mathrm {e v}}, Y ^ {\mathrm {e v}}\right)\right).
$$
The testing procedure is summarized in Phases 2, 3 and 4 in Algorithm 2 and Algorithm 3. In the Higgs experiment, we utilize the Gaussian approximation method to determine the p-values when the witness function  $f$  is not thresholded, which allows us to reach very small p-values and errors under limited computational resource. In cases where the score function  $f$  is thresholded by a value  $t$ , using the Binomial distribution as in Algorithm 3 is more precise and also fast enough.
Given a trained kernel  $K$  trained on  $X^{\mathrm{tr}}$  and  $Y^{\mathrm{tr}}$ , we set  $X^{\mathrm{ev}} = X^{\mathrm{tr}}$  and  $Y^{\mathrm{ev}} = Y^{\mathrm{tr}}$ , and accordingly  $n_{\mathrm{ev}} = n_{\mathrm{tr}}$ . This results in a more efficient use of data (since we reuse  $X^{\mathrm{tr}}$ ,  $Y^{\mathrm{tr}}$  also as  $X^{\mathrm{ev}}$ ,  $Y^{\mathrm{ev}}$ ). Then, out of the untouched portion of the data, we randomly choose  $n_{\mathrm{cal}} = 20,000$  datapoints from both classes to populate  $X^{\mathrm{cal}}$  and  $Y^{\mathrm{cal}}$ , i.e.,  $|X^{\mathrm{cal}}| = |Y^{\mathrm{cal}}| = n_{\mathrm{cal}} = 20,000$ . In addition to the general setting above, for RFM, we need to solve a  $2n_{\mathrm{ev}}$ -dimensional linear equation during inference, which arises from the inverse matrix in Equation (32) (solving  $K(X^{\mathrm{RFM}},X^{\mathrm{RFM}})\pmb {u} = (y^{\mathrm{RFM}})^T$  for  $\pmb {u}\in \mathbb{R}^{2n_{\mathrm{ev}}}$ ). So we set  $n_{\mathrm{ev}} = \min (n_{\mathrm{tr}},10,000)$  that  $X_{\mathrm{ev}},Y_{\mathrm{ev}}$  are randomly sampled from the training set.
In order to compare different benchmarks, we evaluate the expected significance of discovery on a mixture of 1000 backgrounds and 100 signals. For each benchmark and each  $n_{\mathrm{tr}}$ , we train 10 independent models. Then for each trained model we proceed through the Phases 2, 3 (and 4) in Algorithm 2 and Algorithm 3 by 10 times for 10 different ( $X^{\mathrm{ev}}, X^{\mathrm{cal}}, X^{\mathrm{opt}}, Y^{\mathrm{ev}}, Y^{\mathrm{cal}}, Y^{\mathrm{opt}}$ ). The mean and standard deviation from these 100 runs are reported in Figure 8.
We also display in Figure 9 the trade-off  $\mathrm{b}(m, n_{\mathrm{ev}})$  and  $(m, n_{\mathrm{tr}})$  to reach certain levels of significance of discovery in MMD-M. From the bottom left plot, we see that the (averaged) significance is not sensitive to  $n_{\mathrm{ev}}$  when  $\lg n_{\mathrm{ev}}$  is large. So taking  $n_{\mathrm{ev}} = 20,000$  is sufficient.
H.3.2 Evaluating the Error of the Test (4)
We set the parameters to be  $\delta = 0.1$  and  $\pi = \frac{1}{2}\delta$  in our experiments. As explained in Appendix G.3, here we no longer take  $X^{\mathrm{ev}} = X^{\mathrm{tr}}$ . Empirically, taking  $X^{\mathrm{ev}} = X^{\mathrm{tr}}$  yields a very bad threshold  $\gamma(X^{\mathrm{ev}}, Y^{\mathrm{ev}}, \pi)$ . Instead,  $X^{\mathrm{ev}}$  is sampled from untouched datapoints other than  $X^{\mathrm{tr}}$ , and the same applies for  $Y$ . We still take  $n_{\mathrm{ev}} = n_{\mathrm{tr}}$  here, resulting in a total size of  $n_{\mathrm{ev}} + n_{\mathrm{tr}} = 2n_{\mathrm{tr}}$ . Specifically, when  $n_{\mathrm{ev}} \geq 10,000$ , computing a  $n_{\mathrm{ev}} \times n_{\mathrm{ev}}$  Gram matrix becomes computationally expensive, so we adopt Monte Carlo method to compute  $\gamma(X^{\mathrm{ev}}, Y^{\mathrm{ev}}, \pi)$ , in which we subsample 10,000 points from  $X^{\mathrm{ev}}$  and  $Y^{\mathrm{ev}}$  to calculate  $\gamma$  and repeat this process 100 times.
Again, we utilize the Gaussian approximation. Recall that the test is to compare  $T = \frac{1}{m}\sum_{i=1}^{m}f(Z_i)$  with  $\gamma$ . The type 1 and type 2 error are estimated as  $\mathrm{CDF}_{\mathcal{N}(0,1)}\left(-\frac{\gamma(X^{\mathrm{ev}},Y^{\mathrm{ev}},\pi) - \mathbb{E}[f|H_0]}{\sqrt{\mathrm{var}(f|H_0)/m}}\right)$  and
4If the kernel  $K(\cdot ,\cdot) = K_{X^{\mathrm{tr}},Y^{\mathrm{tr}}}(\cdot ,\cdot)$  is independent of  $X^{\mathrm{ev}},Y^{\mathrm{ev}}$  , then we have  $\gamma (X^{\mathrm{ev}},Y^{\mathrm{ev}},\delta /2)\approx \frac{1}{2}\left(\mathbb{E}_{Z\sim P_x}[T(X^{\mathrm{ev}},Y^{\mathrm{ev}},Z)] + \mathbb{E}_{Z\sim \delta P_Y + (1 - \delta)P_X}[T(X^{\mathrm{ev}},Y^{\mathrm{ev}},Z)]\right)$  . However this is no longer true if  $(X^{\mathrm{tr}},Y^{\mathrm{tr}})$  and  $(X^{\mathrm{ev}},Y^{\mathrm{ev}})$  intersect.
$\mathrm{CDF}_{\mathcal{N}(0,1)}\left(-\frac{\mathbb{E}[f|H_1] - \gamma(X^{\mathrm{ev}},Y^{\mathrm{ev}},\pi)}{\sqrt{\operatorname{var}(f|H_1) / m}}\right)$  for the witness function  $f$ , which can be estimated efficiently using the calibration samples  $X^{\mathrm{cal}}$ ,  $Y^{\mathrm{cal}}$ .
We consider both the regimes of fixing kernels and varying kernels (training kernel based on  $n$ ). The results are shown in the top plot in Figure 1.2 and the top plot in Figure 9. For each point on the plot, we train 30 independent models and test each model 10 times, and report the average of these 300 runs. In both plots, we observe the asymmetric  $m$  vs  $n$  trade-off.
Algorithm 2 Estimate the significance of discovery of an input  $Z_{\mathrm{test}}$ , using the original statistic
Input:  $(X^{\mathrm{tr}}, X^{\mathrm{ev}}, X^{\mathrm{cal}})$ ,  $(Y^{\mathrm{tr}}, Y^{\mathrm{ev}}, Y^{\mathrm{cal}})$ ; parametrized kernel  $K_{\omega}$ ; input  $Z_{\mathrm{test}}$ .  
# Phase 1: Kernel training on  $X^{\mathrm{tr}}$  and  $Y^{\mathrm{tr}}$ $\omega \leftarrow \arg \max_{\omega}^{\text {optimizer }} \hat{J}(X^{\mathrm{tr}}, Y^{\mathrm{tr}}; K_{w})$  # maximize objective  $\hat{J}(X^{\mathrm{tr}}, Y^{\mathrm{tr}}; K_{\omega})$  as in (8)  
# Phase 2: Distributional calibration of test statistic  
Scores $^{(0)}$ $\leftarrow f(X^{\mathrm{cal}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}})$  # Scores $^{(0)}$  has a length of  $n_{\mathrm{cal}}$   
Scores $^{(1)}$ $\leftarrow f(Y^{\mathrm{cal}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}})$  # Scores $^{(1)}$  has a length of  $n_{\mathrm{cal}}$ $\theta_0 \leftarrow \text{mean}(\text{Scores}^{(0)})$  # estimate  $\mathbb{E}[f(Z)|Z \sim P_X]$ $\theta_1 \leftarrow \text{mean}(\text{Scores}^{(1)})$  # estimate  $\mathbb{E}[f(Z)|Z \sim P_Y]$ $\sigma_0 \leftarrow \text{std}(\text{Scores}^{(0)})$  # estimate  $\sqrt{\text{var}[f(Z)|Z \sim P_X]}$   
# Phase 3: Inference with input  $Z_{\text{test}}$ $m \leftarrow \text{length}(Z_{\text{test}})$ $T \leftarrow T_f(Z_{\text{test}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}}) = \text{mean}(f(Z_{\text{test}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}}))$  # compute test statistic  
 $Z_{\text{discovery}} \leftarrow \frac{T - \theta_0}{\sigma_0 / \sqrt{m}}$   
Output: Estimated significance:  $Z_{\text{discovery}}$
Algorithm 3 Estimate the significance of discovery of an input  $Z_{\mathrm{test}}$ , applying the thresholding trick
Input:  $(X^{\mathrm{tr}}, X^{\mathrm{ev}}, X^{\mathrm{cal}}, X^{\mathrm{opt}})$ ,  $(Y^{\mathrm{tr}}, Y^{\mathrm{ev}}, Y^{\mathrm{cal}}, Y^{\mathrm{opt}})$ ; parametrized kernel  $K_{\omega}$ ; input  $Z_{\mathrm{test}}$ .  
# Phase 1: Kernel training on  $X^{\mathrm{tr}}$  and  $Y^{\mathrm{tr}}$ $\omega \leftarrow \arg \max_{\omega}^{\text {optimizer }} \hat{J}(X^{\mathrm{tr}}, Y^{\mathrm{tr}}; K_w)$  # maximize objective  $\hat{J}(X^{\mathrm{tr}}, Y^{\mathrm{tr}}; K_{\omega})$  as in (8)  
# Phase 2: Find the best threshold  
 $\text { Scores }^{(0)} \leftarrow f(X^{\mathrm{opt}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}})$ $\text { Scores }^{(1)} \leftarrow f(Y^{\mathrm{opt}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}})$  # witness function as in (33)  
for  $i = 1, 2, \ldots, 2n_{\mathrm{opt}}$  do  
 $t = (\text { Scores }^{(0)} \cup \text { Scores }^{(1)})[i]$   
TP, TN = mean(Scores  $^{(1)} > t$ ), mean(Scores  $^{(0)} < t$  ) # true positive and true negative rate  
power_i =  $\frac{\text { TP + TN - 1 } }{\sqrt{\text { TN } (1 - \text { TN })}}$  # find t to maximize the (estimated) p-value  
end for  
 $t_{\mathrm{opt}} = (\text { Scores }^{(0)} \cup \text { Scores }^{(1)})[\arg \max_{i} \text { power}_i]$   
# Phase 3: Distributional calibration of test statistic (under null hypothesis)  
 $\text { Scores }^{(0)} \leftarrow (f(X^{\mathrm{cal}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}}) > t)$  # Scores  $^{(0)} \in \{0, 1\}^{n_{\mathrm{ev}}}$ $\text { Scores }^{(1)} \leftarrow (f(Y^{\mathrm{cal}}; X^{\mathrm{ev}}, Y^{\mathrm{ev}}) > t)$  # Scores  $^{(1)} \in \{0, 1\}^{n_{\mathrm{ev}}}$ $\theta_0 \leftarrow \text { mean(Scores }^{(0)})$  # estimate  $\mathbb{E}[f_t(Z)|Z \sim P_X] \in [0, 1]$ $\theta_1 \leftarrow \text { mean(Scores }^{(1)})$  # estimate  $\mathbb{E}[f_t(Z)|Z \sim P_Y] \in [0, 1]$   
# Phase 4: Inference with input  $Z_{\text {test }}$   
m ← length(Z_test)  
T ← T_f(Z_test; X^ev, Y^ev) = mean(f(Z_test; X^ev, Y^ev) > t) # compute test statistic  
Z_discovery ← CDF^{-1}_{\mathcal{N}(0, 1)}(CDF_{Bin(m, \theta_0)}(T))  
Output: Estimated significance: Z_discovery
Figure 8: Complete image of Figure 1.2 in the main text. The mean and standard deviation are calculated based on 100 runs. See Appendix H for details.
Figure 9: The top plot displays the  $(m, n_{\mathrm{ev}})$  trade-off to reach certain levels of total error using  $n_{\mathrm{tr}} = 1.3 \times 10^{6}$  in MMD-M. The bottom figures show the trade-off of  $(m, n_{\mathrm{ev}})$  and  $(m, n_{\mathrm{tr}})$  to reach certain level of significance of discovery in MMD-M. In the bottom left figure, we fix  $n_{\mathrm{tr}} = 1.3 \times 10^{6}$ . In the bottom right figure, we fix  $n_{\mathrm{ev}} = 20,000$ . See Appendix H for details.
I Limitations and Future Directions
Finally, we discuss several limitations of our work and raise open questions that we hope will be addressed in future works. From the theoretical side of our arguments, we point out several aspects. First, our upper bound (on the minimax sample complexity) Theorem 3.1 has a likely sub-optimal dependence on  $\alpha ,\delta$  . Second, it might be possible to improve our lower bound to a more natural form by replacing  $\| \lambda \|_{2,J_{\epsilon}^{\star}}$  by  $\| \lambda \| _2$  and removing the constraint that the top eigenfunction has to be constant. Third, it remains open to extend our theory to include data-dependent  $K$  , as opposed to fixed  $K$
Empirically, our proposal Algorithm 1 can be inefficient in Phase 2 (prior works such as [39] have used permutation-based arguments for a more efficient estimate), which we adopted due to its simplicity and universality in all benchmarks. Moreover, one might hope that LFHT/mLFHT can be extended to more complex applications, such as text data or videos. Such questions are important to investigate as a future direction.