SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| X - X ^ {\prime} \right\| _ {2} ^ {2} \right] = 2 \mathbb {E} \left[ \left\| X \right\| _ {2} ^ {2} \right] = 4 d, \\ \mathbb {E} \left[ \| Y - Y ^ {\prime} \| _ {2} ^ {2} \right] = 2 \mathbb {E} \left[ \| Y \| _ {2} ^ {2} \right] = 2 \mathrm {t r} (B _ {s} B _ {s} ^ {\top} + I _ {d}) = 2 (d + \sum_ {i = 1} ^ {d} \beta_ {i} ^ {2}). \\ \end{array}
$$
4. Gaussian model. We have
$$
\begin{array}{l} \mathbb {E} \left[ (X - X ^ {\prime}) ^ {2} \right] = 2 \mathbb {E} \left[ X ^ {2} \right] = 2, \\ \mathbb {E} \left[ (Y - Y ^ {\prime}) ^ {2} \right] = 2 \mathbb {E} \left[ Y ^ {2} \right] = 2 \left(1 + \beta^ {2}\right). \\ \end{array}
$$
Ridge Regression. We use ridge regression as an underlying predictive model:  $\hat{g}_t(x) = \beta_0^{(t)} + x\beta_1^{(t)}$  where the coefficients are obtained by solving:
Let  $\Gamma = \mathrm{diag}(0,1)$ . Let  $\mathbf{X}_{t - 1}\in \mathbb{R}^{2(t - 1)\times 2}$  be such that  $(\mathbf{X}_{t - 1})_i = (1,X_i)$ ,  $i\in [1,2(t - 1)]$ . Finally, let  $\mathbf{Y}_{t - 1}$  be a vector of responses:  $(\mathbf{Y}_{t - 1})_i = Y_i$ ,  $i\in [1,2(t - 1)]$ . Then:
$$
\boldsymbol {\beta} ^ {(t)} = \underset {\boldsymbol {\beta}} {\arg \min} \left\| \mathbf {Y} _ {t - 1} - \mathbf {X} _ {t - 1} \boldsymbol {\beta} \right\| ^ {2} + \lambda \boldsymbol {\beta} ^ {\top} \Gamma \boldsymbol {\beta} = \left(\mathbf {X} _ {t - 1} ^ {\top} \mathbf {X} _ {t - 1} + \lambda \Gamma\right) ^ {- 1} \left(\mathbf {X} _ {t - 1} ^ {\top} \mathbf {Y} _ {t - 1}\right).
$$
E.2 Additional Experiments for Seq-C-IT
In Figure 7, we present average stopping times for ITs under the synthetic settings from Section 3. We confirm that all tests adapt to the complexity of a problem at hand, stopping earlier on easy tasks and later on harder ones. We also consider two additional synthetic examples where Seq-C-IT outperforms a kernelized approach:
1. Sparse signal model. Let  $(X_{t})_{t\geq 1}$  and  $(\varepsilon_{t})_{t\geq 1}$  be two independent sequences of standard Gaussian random vectors in  $\mathbb{R}^d$ :  $X_{t},\varepsilon_{t}\stackrel {\mathrm{iid}}{\sim}\mathcal{N}(0,\mathbf{I}_{d}),t\geq 1$ . We take
$$
\left(X _ {t}, Y _ {t}\right) = \left(X _ {t}, B _ {s} X _ {t} + \varepsilon_ {t}\right),
$$
where  $B_{s} = \mathrm{diag}(\beta_{1},\dots ,\beta_{d})$  and only  $s = 5$  of  $\{\beta_i\}_{i = 1}^d$  are nonzero being sampled from Unif([-0.5,0.5]). We consider  $d\in \{5,\ldots ,50\}$ .
(b) HTDD model.
Figure 7: Stopping times of ITs on synthetic data from Section 3. Subplot (a) shows that SKIT is only marginally better than Seq-C-IT (MLP) due to slightly better sample efficiency under the spherical model (no localized dependence). Under the structured HTDD model, SKIT is inferior to Seq-C-ITs.
2. Nested circles model. Let  $(L_{t})_{t\geq 1}$ ,  $(\Theta_{t})_{t\geq 1}$ ,  $(\varepsilon_{t}^{(1)})_{t\geq 1}$ ,  $(\varepsilon_{t}^{(2)})_{t\geq 1}$  denote sequences of random variables where  $L\stackrel{\mathrm{iid}}{\sim}\mathrm{Unif}(1,\ldots,l)$  for some prespecified  $l\in \mathbb{N}$ ,  $\Theta_t\stackrel{\mathrm{iid}}{\sim}\mathrm{Unif}([0,2\pi])$  and  $\varepsilon_t^{(1)},\varepsilon_t^{(2)}\stackrel{\mathrm{iid}}{\sim}\mathcal{N}(0,(1/4)^2)$ . For  $t\geq 1$ , we take
$$
\left(X _ {t}, Y _ {t}\right) = \left(L _ {t} \cos \left(\Theta_ {t}\right) + \varepsilon_ {t} ^ {(1)}, L _ {t} \sin \left(\Theta_ {t}\right) + \varepsilon_ {t} ^ {(2)}\right). \tag {67}
$$
We consider  $l \in \{1, \dots, 10\}$ .
In Figure 8, we show that Seq-C-ITs significantly outperform SKIT under these models. We note that the degrading performance of kernel-based tests under the nested circles model (67) has been also observed in earlier works [Berrett and Samworth, 2019, Podkopaev et al., 2023].
E.3 Sequential vs Batch 2STs
The role of sequential tests is to complement existing batch tests and not replace those. Generally, if one is ready to commit to a particular sample size, it is better to refer to batch tests. To understand the loss of power, we conducted an experiment where we take  $P = \mathcal{N}(0,1)$ ,  $Q = \mathcal{N}(\delta, 1)$  and use  $k$ -NN predictor as an underlying predictor. As we point out in Remark 1, if an analyst stops the experiment early, i.e., before the wealth exceeds  $1 / \alpha$ , there is one modification that allows using non-exhausted type-I error budget to strictly improve power [Ramdas and Manole, 2023]: one may use a different threshold for rejecting the null, namely  $U / \alpha$ , where  $U$  is an independently drawn (stochastically larger than) uniform random variable on  $[0,1]$ .
We set the sample size to 1500 and compared three tests: batch (with half of the data used for training and half for inference), Seq-C-2ST (our sequential test), and Seq-C-2ST + Randomization (as per Remark 1). In Figure 9, we observe that the batch test has only slightly higher power than our sequential one when the sample size is specified beforehand. Yet in cases where the power is less than one using a sequential test allows collecting more data to improve it, but with the batch test, nothing can be done since the type-1 error budget is fully exhausted.
(b) Sparse signal model.
(a) Sparse signal model.
(c) Nested circles model.
Figure 8: Rejection rates (left column) and average stopping times (right column) of sequential ITs for synthetic datasets from Appendix E.2. In both cases, SKIT is inferior to Seq-C-ITs.
(d) Nested circles model.
Figure 9: Comparison between the batch and sequential 2STs on Gaussian data when the sample size is set beforehand to 1500. First, invoking randomization improves the power of our sequential test across all settings. Second, batch 2ST has only slightly higher power across many settings. Note that the power of Seq-C-2ST can be easily improved after collecting a bit more data, yet this is not allowed for the batch test.