SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(| \hat {w} _ {i} ^ {n} - w _ {i} ^ {*} | > \epsilon\right) \\ \leq \mathbb {P} (\exp (- a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2}) + | Z _ {i, n} | \exp [ - a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2} + \sqrt {2} a _ {i} | Z _ {i, n} | / \hat {\sigma} _ {n, - i} (x _ {i}) ] > \epsilon) \\ = \mathbb {P} \left( \right.\exp \left(- a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} \left(x _ {i}\right) ^ {2}\right) + \left| Z _ {i, n} \right| \exp \left[ - a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} \left(x _ {i}\right) ^ {2} + \sqrt {2} a _ {i} \left| Z _ {i, n} \right| / \hat {\sigma} _ {n, - i} \left(x _ {i}\right)\right] > \epsilon , \\ \left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| > a _ {i} / (2 \sqrt {2})) + \\ \mathbb {P} \left( \right.\exp \left(- a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2}\right) + | Z _ {i, n} | \exp \left[ - a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2} + \sqrt {2} a _ {i} | Z _ {i, n} | / \hat {\sigma} _ {n, - i} (x _ {i}) \right] > \epsilon , \\ \left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| \leq a _ {i} / (2 \sqrt {2})) \\ \leq \mathbb {P} \left(\left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| > a _ {i} / (2 \sqrt {2})\right) + \\ \mathbb {P} \left(\exp \left(- a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2}\right) + | Z _ {i, n} | \exp \left[ - a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} (x _ {i}) ^ {2} + \sqrt {2} a _ {i} | Z _ {i, n} | / \hat {\sigma} _ {n, - i} (x _ {i}) \right] > \epsilon , \right. \\ \left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| \leq a _ {i} / (2 \sqrt {2})) \\ \leq \mathbb {P} \left(\left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| > a _ {i} / (2 \sqrt {2})\right) + \\ \mathbb {P} \left(\left(1 + \frac {a _ {i}}{2 \sqrt {2} \hat {\sigma} _ {n , - i} (x _ {i})}\right) \exp \left(\frac {- a _ {i} ^ {2}}{2 \hat {\sigma} _ {n , - i} (x _ {i}) ^ {2}}\right) > \epsilon , | Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) | \leq a _ {i} / (2 \sqrt {2})\right) \\ \leq \mathbb {P} \left(\left| Z _ {i, n} \hat {\sigma} _ {n, - i} \left(x _ {i}\right) \right| > a _ {i} / (2 \sqrt {2})\right) + \mathbb {P} \left(\left(1 + \frac {a _ {i}}{2 \sqrt {2} \hat {\sigma} _ {n , - i} \left(x _ {i}\right)}\right) \exp \left(\frac {- a _ {i} ^ {2}}{2 \hat {\sigma} _ {n , - i} \left(x _ {i}\right) ^ {2}}\right) > \epsilon\right) \tag {7} \\ \end{array}
$$
where the second-to-last inequality follows by substituting  $|Z_{i,n}\hat{\sigma}_{n, - i}(x_i)|\leq a_i / (2\sqrt{2})$  into the bound and algebraically simplifying, and the last inequality follows from (5). Now, the first term on the right converges to 0 since  $Z_{i,n}\hat{\sigma}_{n, - i}(x_i) = \tau (x_i) - \hat{\tau}_{n, - i}(x_i)\stackrel {p}{\to}0.$  We examine the second term. Define  $\xi_{i,n} = a_i / (\sqrt{2}\hat{\sigma}_{n, - i}(x_i))$  . Since a convex function is no smaller than its tangent line, we have  $(1 + \xi_{i,n} / 2)\leq \exp (\xi_{i,n} / 2)$  . Then,
$$
\begin{array}{l} \mathbb {P} \left(\left(1 + \frac {a _ {i}}{2 \sqrt {2} \hat {\sigma} _ {n , - i} (x _ {i})}\right) \exp \left(\frac {- a _ {i} ^ {2}}{2 \hat {\sigma} _ {n , - i} (x _ {i}) ^ {2}}\right) > \epsilon\right) \leq \mathbb {P} \big ((1 + \xi_ {i, n} / 2) \exp (- \xi_ {i, n} ^ {2}) > \epsilon \big) \\ \leq \mathbb {P} \left(\exp \left(- \xi_ {i, n} ^ {2} + \xi_ {i, n} / 2\right) > \epsilon\right) \\ \end{array}
$$
which converges to 0 as  $\hat{\sigma}_{n, -i}(x_i) \stackrel{p}{\to} 0$  (and thus  $\xi_{i,n}$  diverges in probability to  $\infty$ ). Thus, we have shown that
$$
\mathbb {P} \left(\left| \hat {w} _ {i} ^ {n} - w _ {i} ^ {*} \right| > \epsilon\right)\rightarrow 0,
$$
which establishes the desired consistency.
E.3 Proof of Thm. 3.3: FLASH limits unnecessary stopping
Suppose that no participant group is harmed so that  $w_{i}^{*} = 0$  for all  $i$ . Let  $a = \delta / \sqrt{2}$ , and  $a_{i} = |\tau(x_{i}) - \delta| / \sqrt{2}$ , and note that  $a \leq a_{i}$  for each  $i$  as each  $\tau(x_{i}) \leq 0$ . Thus, if  $|Z_{i,n}\hat{\sigma}_{n,-i}(x_{i})| \leq a / (2\sqrt{2})$ , then
$|Z_{i,n}\hat{\sigma}_{n, - i}(x_i)| \leq a_i / (2\sqrt{2})$  and hence, as in the proof of Thm. 3.2,
$$
\hat {w} _ {i} ^ {n} \leq \left(1 + \frac {a _ {i}}{2 \sqrt {2} \hat {\sigma} _ {n , - i} (x _ {i})}\right) \exp \left(\frac {- a _ {i} ^ {2}}{2 \hat {\sigma} _ {n , - i} (x _ {i}) ^ {2}}\right).
$$
Fix any  $\epsilon > 0$ , let  $\xi_{i,n} = a_i / (\sqrt{2}\hat{\sigma}_{n, -i}(x_i))$ , and define  $f(x) = (1 + x / 2)\exp (-x^2)$ . As we did for equation (7) in the proof of Thm. 3.2, we apply the law of total probability and equation (5) to write,
$$
\begin{array}{l} \mathbb {P} (\sum_ {i} \hat {w} _ {i} ^ {n} > \epsilon) \leq \mathbb {P} (\max _ {i \leq n} | Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) | > a / (2 \sqrt {2})) + \\ \mathbb {P} \left(\sum_ {i} \hat {w} _ {i} ^ {n} > \epsilon , \max _ {i \leq n} | Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) | \leq a / (2 \sqrt {2})\right) \\ \leq \mathbb {P} \left(\max  _ {i \leq n} \left| Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) \right| > a / (2 \sqrt {2})\right) + \\ \mathbb {P} \left(\sum_ {i} \left(1 + a _ {i} / \left(2 \sqrt {2} \hat {\sigma} _ {n, - i} \left(x _ {i}\right)\right)\right) \exp \left(- a _ {i} ^ {2} / \left(2 \hat {\sigma} _ {n, - i} \left(x _ {i}\right) ^ {2}\right)\right) > \epsilon\right) \\ = \mathbb {P} \left(\max  _ {i \leq n} \left| Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) \right| > a / (2 \sqrt {2})\right) + \mathbb {P} \left(\sum_ {i} f (\xi_ {i, n}) > \epsilon\right). \\ \end{array}
$$
Since  $\max_{i\leq n}|Z_{i,n}\hat{\sigma}_{n, - i}(x_i)| = o_p(1)$  by assumption, we have  $\mathbb{P}(\max_{i\leq n}|Z_{i,n}\hat{\sigma}_{n, - i}(x_i)| > a / (2\sqrt{2}))\to 0$  . Meanwhile, since a convex function is no smaller than its tangent line, we have  $(1 + x / 2)\leq \exp (x / 2)$  and hence
$$
f (x) \leq \exp (- x ^ {2} + x / 2) \leq \exp (- x ^ {2} + x ^ {2} / 2 + 1 / 8) = \exp (- x ^ {2} / 2 + 1 / 8)
$$
where we used the arithmetic-geometric mean inequality in the final inequality. Therefore, since  $a \leq \min_i a_i$ ,
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {i} f \left(\xi_ {i, n}\right) > \epsilon\right) \leq \mathbb {P} \left(\sum_ {i} \exp \left(- \xi_ {i, n} ^ {2} / 2 + 1 / 8\right) > \epsilon\right) \\ \leq \mathbb {P} \left(n \exp \left(- \left(\min  _ {i \leq n} \xi_ {i, n} ^ {2}\right) / 2 + 1 / 8\right) > \epsilon\right) \\ \leq \mathbb {P} \left(n \exp \left(- \left(\min  _ {i \leq n} a _ {i} ^ {2} / \hat {\sigma} _ {n, - i} ^ {2} \left(x _ {i}\right)\right) / 4 + 1 / 8\right) > \epsilon\right) \\ \leq \mathbb {P} \left(\exp \left(- a ^ {2} / \left(4 \max  _ {i \leq n} \hat {\sigma} _ {n, - i} ^ {2} \left(x _ {i}\right)\right) + 1 / 8 + \log n\right) > \epsilon\right) \\ \end{array}
$$
Since  $\max_{i\leq n}\hat{\sigma}_{n, - i}^2 (x_i) = o_p(1 / \log (n))$  by assumption, we further have  $\mathbb{P}(\sum_i f(\xi_{i,n}) > \epsilon)\to 0$  Since  $\epsilon >0$  was arbitrary, we have shown that  $\sum_{i = 1}^{n}\hat{w}_i^n\xrightarrow{p}0$
Now, recall the form of the CLASH weighted z-statistic
$$
\lambda_ {n} ^ {w} = \frac {\sqrt {\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n}}}{\sqrt {2 \sigma^ {2}}} \left(\frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} y _ {i} d _ {i}}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} d _ {i}} - \frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} y _ {i} (1 - d _ {i})}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} (1 - d _ {i})}\right)
$$
Define  $c$  such that  $|y_i| \leq c$ . We know  $c$  must exist, since the outcomes are bounded by assumption. Then, we have that
$$
\begin{array}{l} | \lambda_ {n} ^ {w} | = \left| \frac {\sqrt {\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n}}}{\sqrt {2 \sigma^ {2}}} \left(\frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} y _ {i} d _ {i}}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} d _ {i}} - \frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} y _ {i} (1 - d _ {i})}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} (1 - d _ {i})}\right) \right| \\ \leq \frac {\sqrt {\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n}}}{\sqrt {2 \sigma^ {2}}} \left(\frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} c d _ {i}}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} d _ {i}} - \frac {\sum_ {i = 1} ^ {n} \hat {w} _ {i} ^ {n} (- c) (1 - d _ {i})}{\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n} (1 - d _ {i})}\right) \\ = \frac {\sqrt {\sum_ {1 = 1} ^ {n} \hat {w} _ {i} ^ {n}}}{\sqrt {2 \sigma^ {2}}} 2 c \\ \xrightarrow {p} 0 \\ \end{array}
$$
Thus, we see that the weighted test statistic  $\lambda_{n}^{w}$  converges in probability to 0. Now, the test can only reject if  $\lambda_{n}^{w}$  exceeds a fixed and positive bound  $b_{\alpha}$ . By the definition of convergence in probability, this probability must shrink to zero.
F Additional Theoretical Results
Corollary F.1 (CLASH weights converge quickly to optimal weights). Assume that  $\hat{\sigma}_{n, -i}^2(x_i) = o_p(1 / \log(n))$ . Then, Thm. 3.2 implies that  $|\hat{w}_i^n - w_i^*| = o_p(1 / n)$ .
Proof. Define  $a_i = |\tau(x_i) - \delta| / \sqrt{2}$  and  $Z_{i,n} = \frac{\tau(x_i) - \hat{\tau}_{n, -i}(x_i)}{\hat{\sigma}_{n, -i}(x_i)}$ . Fix any  $\epsilon > 0$ , let  $\xi_{i,n} = a_i / (\sqrt{2}\hat{\sigma}_{n, -i}(x_i))$ , and define  $f(x) = (1 + x/2)\exp(-x^2)$ . Then, as in equation (7) in the proof of Thm. 3.2, we apply the law of total probability and equation (5) to yield
$$
\begin{array}{l} \mathbb {P} \left(n \left| \hat {w} _ {i} ^ {n} - w _ {i} ^ {*} \right| > \epsilon\right) \leq \mathbb {P} \left(\left| Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) \right| > a _ {i} / (2 \sqrt {2})\right) + \\ \mathbb {P} \left(n \left(1 + \frac {a _ {i}}{2 \sqrt {2} \hat {\sigma} _ {n , - i} (x _ {i})}\right) \exp \left(\frac {- a _ {i} ^ {2}}{2 \hat {\sigma} _ {n , - i} (x _ {i}) ^ {2}}\right) > \epsilon\right) \\ = \mathbb {P} \left(\left| Z _ {i, n} \hat {\sigma} _ {n, - i} (x _ {i}) \right| > a _ {i} / (2 \sqrt {2})\right) + \mathbb {P} \left(n f (\xi_ {i, n}) > \epsilon\right). \\ \end{array}
$$
The first term on the right converges to 0 since  $Z_{i,n}\hat{\sigma}_{n, - i}(x_i) = \tau (x_i) - \hat{\tau}_{n, - i}(x_i)\overset {p}{\to}0$  (by the assumptions of Thm. 3.2). We focus on the second term. Since a convex function is no smaller than its tangent line, we have  $(1 + \xi_{i,n} / 2)\leq \exp (\xi_{i,n} / 2)$  and hence
$$
f \left(\xi_ {i, n}\right) \leq \exp \left(- \xi_ {i, n} ^ {2} + \xi_ {i, n} / 2\right) \leq \exp \left(- \xi_ {i, n} ^ {2} + \xi_ {i, n} ^ {2} / 2 + 1 / 8\right) = \exp \left(- \xi_ {i, n} ^ {2} / 2 + 1 / 8\right)
$$
where we used the arithmetic-geometric mean inequality in the final inequality. Thus,
$$
\begin{array}{l} \mathbb {P} \left(n f \left(\xi_ {i, n}\right) > \epsilon\right) \leq \mathbb {P} \left(n \exp \left(- \xi_ {i, n} ^ {2} / 2 + 1 / 8\right) > \epsilon\right) \\ \leq \mathbb {P} \left(\exp \left(- a _ {i} ^ {2} / 4 \hat {\sigma} _ {n, - i} ^ {2} + 1 / 8 + \log n\right) > \epsilon\right) \\ \rightarrow 0, \\ \end{array}
$$
since  $\hat{\sigma}_{n, -i}(x_i)^2 = o_p(1 / \log (n))$ . Thus,  $\mathbb{P}(n|\hat{w}_i^n - w_i^*| > \epsilon) \to 0$ , and so by definition,  $|\hat{w}_i^n - w_i^*| = o_p(1/n)$ .
Theorem F.2 (CLASH limits unnecessary stopping for the Gaussian SPRT). Consider a stopping test with the Gaussian SPRT and weights estimated using CLASH. If  $\max_{t\leq n}\hat{\sigma}_{n, - t}(x_t)^2 = o_p(1 / \log (n))$ ,  $\max_{t\leq n}|\tau (x_t) - \hat{\tau}_{n, - i}(x_t)| = o_p(1)$ , and  $z_{t}$  are uniformly bounded, then the stopping probability of the test converges to zero if no participant group is harmed.
Proof. We adopt the setup for the SPRT described in App. B. Recall that the Gaussian SPRT test statistic is given by
$$
\lambda_ {2 n} = \beta \sum_ {t = 1} ^ {n} z _ {t} - \frac {n}{2} \beta^ {2}
$$
and the CLASH- weighted version of this statistic is given by
$$
\lambda_ {2 n} ^ {w} = \beta \sum_ {t = 1} ^ {n} \hat {w} _ {t} ^ {n} z _ {t} - \frac {\sum_ {t = 1} ^ {n} \hat {w} _ {t} ^ {n}}{2} \beta^ {2}.
$$
Now, by assumption, there exists some  $c$  that bounds  $|z_{t}|$  for all  $t$ . Since no participant is harmed,  $w_{i}^{*} = 0$  for all  $i$ . Similar to the proof in App. E.3, if  $\max_{i\leq n}\hat{\sigma}_{n, - i}(x_i)^2 = o_p(1 / \log (n))$  and  $\max_{i\leq n}|\tau (x_i) - \hat{\tau}_{n, - i}(x_i)| = o_p(1)$ , then  $\sum_{t = 1}^{n}\hat{w}_t^n\to 0$ . Thus, since  $z_{t}$  are uniformly bounded by some  $c$ ,
$$
\begin{array}{l} \lambda_ {2 n} ^ {w} \leq \beta \sum_ {t = 1} ^ {n} \hat {w} _ {t} ^ {n} c \\ \xrightarrow {p} 0 \\ \end{array}
$$
This shows that  $\lambda_{2n}^{w}\xrightarrow{p}0$  as  $n\to \infty$ . Thus,  $\mathbb{P}(\lambda_{2n}^w >\log \alpha)\to 0$  for any fixed  $\alpha >0$ , proving the claim.
G Stopping Only on the Harmed Group
If the investigators choose to stop the experiment only on the harmed group, they face two practical challenges: identifying the harmed group and ATE estimation. We discuss each of these below.
Identifying the harmed group. To stop the experiment only for harmed participants, investigators must first identify the harmed group. This is non-trivial, since group membership is unobserved. The distribution of FLASH weights at stopping time can help in this task: groups with estimated weights close to 1 are likely to be harmed. Fig. S19 illustrates this in our empirical application: the estimated weights in Regions 1 and 2 are both close to 1, indicating that these are the groups on which to stop. With few covariates, investigators can manually inspect the weight distribution for each covariate combination to identify the harmed group. With many covariates, investigators can use a simple heuristic: a regression decision tree on the estimated FLASH weights can find the covariate values for which the weights are the largest. Limiting the depth of this tree can ensure that the identified group is actionable (i.e., it is possible to stop the experiment on it) and of non-trivial size.
There are alternative approaches to this harmed group identification task; for example, investigators can use subgroup identification methods on the raw outcomes (e.g. [1, 30, 44]). We are agnostic to the choice of method: investigators can pick the approach most appropriate for their domain.
ATE estimation. Stopping the experiment on only one group can affect inference at the end of the experiment, as the treatment is no longer randomly assigned across covariates. To estimate the whole population ATE, we recommend using inverse propensity weights (IPW) to correct for the induced selection bias. Let  $\tilde{g}$  be the group that the experiment is stopped on and  $p(\tilde{g})$  denote the proportion of the total population that comes from  $\tilde{g}$ .  $p(\tilde{g})$  may either be known (e.g., from domain knowledge or prior experiments) or estimated from the interim data when the experiment is stopped on  $\tilde{g}$ . Let  $p_{collected}(\tilde{g})$  denote the observed proportion of  $\tilde{g}$  in the data collected over the entire experiment (i.e., pre- and post-stopping on  $\tilde{g}$ ). Then, to estimate the whole-population ATE, observations from  $\tilde{g}$  should be assigned a weight of  $p(\tilde{g}) / p_{collected}(\tilde{g})$  and all other observations should be assigned a weight of 1. This IPW approach will lead to unbiased ATE inference (see the example in Tab. S6).
H Additional Simulation Experiments
In this section, we present detailed results from our simulation experiments. App. H.1 presents results with Gaussian outcomes and App. H.2 presents results with time-to-event outcomes. In almost all settings, FLASH outperforms both the homogeneous and SUBTLE baselines. We note that FLASH's performance gains are relatively limited in three situations: (1) when the harmed group forms  $50\%$  of the population (i.e., is no longer a true "minority") and the treatment has no effect on the remaining population, (2) when the experiment collects a very large number of covariates  $(d = 500)$ , and (3) when the experiment recruits a very small number of participants  $(N = 200)$ . However, even in these settings, FLASH's stopping probability is equivalent to the baselines.
H.1 Gaussian Outcomes
H.1.1 Setup
The simulation setup is the same as described in Sec. 4; we include it here again for easy reference. We consider a randomized experiment that evaluates the effect of  $D$  on  $Y$ . Participants come from two groups, with  $G \in \{0,1\}$  indicating membership in a minority group. We do not observe  $G$ , but observe a set of binary covariates  $X = [X_1,..,X_d]$ , where  $d$  varies between 3 and 10 and  $p(X_j = 1) = 0.5 \forall j$ .  $X$  maps deterministically to  $G$ , with  $G = \prod_{j=1}^{k} X_j$ . We vary  $k$  between 1 and 3: the expected size of the minority thus varies between  $12.5\%$  and  $50\%$  (of all participants).  $Y$  is normally distributed, with  $Y|G = 0 \sim \mathcal{N}(\theta_0D,1)$  and  $Y|G = 1 \sim \mathcal{N}(\theta_1D,1)$ . We vary  $\theta_0$  between 0 and -0.1; the majority group is thus unaffected or benefited by the treatment. We vary  $\theta_1$  between 0 and 1: the minority group is thus either unaffected or harmed.
The experiment runs for  $N = 4000$  participants and  $T = 2000$  time steps, recruiting one treated and one untreated participant at each step. The experiment has three checkpoints, with 1,000, 2,000, and 3,000 participants (corresponding to  $25\%$ ,  $50\%$ , and  $75\%$  of the total time duration). At each checkpoint, we run FLASH and compute its stopping probability across 1,000 replications. In Stage 1, FLASH uses a causal forest with 5-fold CV and  $\delta = 0.1$ . Standard errors were estimated by the causal forest itself (i.e., not via the bootstrap). In Stage 2, FLASH uses one of four commonly-used stopping tests: an OF-adjusted z-test, an mSPRT [14], a MaxSPRT [16], and a Bayesian estimation-based test.
Each replication uses its own random seed. One replication—including all combinations of effect size, minority group size, number of covariates and stopping tests—takes between 58 and 82 minutes to run on a single CPU (depending on the random seed). This yields a total compute time of approximately 1,200 hours for the Gaussian experiments. The simulations were run in parallel on an academic computing cluster with over 200 CPUs. Note that each replication used one CPU with 4GB of RAM.
H.1.2 Impact of Hyperparameter  $\delta$
We first discuss the choice of hyperparameter  $\delta$ . We recommend that investigators set this value to the minimum effect size of interest (MESI). Here, we evaluate three different values of the MESI: 0.05, 0.1, and 0.2. Note that an effect size of 0.05 is detectable in this experiment (with  $N = 4,000$ ) with  $70\%$  power; we thus do not consider  $\mathrm{MESI} < 0.05$ , as such small effects would not be reliably detectable.
We present our results for FLASH with an OF-adjusted z-test and an mSPRT in Fig. S5. In most scenarios, all three settings of  $\delta$  perform similarly. For the mSPRT with a  $12.5\%$  minority group, we note that a larger value of  $\delta$  corresponds with higher stopping probability for larger effects. This is intuitive, as the higher  $\delta$  is, the more weight FLASH will assign to parts of the covariate space that show clear signs of harm. However, FLASH performs better than the homogeneous and SUBTLE baselines for all three values of  $\delta$ . Thus, the results presented in the main text are robust to an increase or decrease in this hyperparameter.
(a) O'Brien Fleming
(b) mSPRT
Figure S5: Effect of  $\delta$  on CLASH's stopping probability. We consider the stopping probability of an (a) OF-adjusted z-test and (b) mSPRT across 1,000 replications of an experiment with Gaussian outcomes,  $N = 4000$ , and five covariates.
H.1.3 Larger Minority Group
Fig. 2 in the main text summarizes FLASH's performance in an experiment where the minority group comprises  $12.5\%$  of the population. We now increase the size of the minority group, and present results in Fig. S6. With a  $25\%$  minority group, FLASH improves stopping probability over the homogeneous baseline, though the gains are more modest than for a  $12.5\%$  minority group. FLASH also performs slightly better than SUBTLE, though this difference is also reduced. All methods perform similarly with a  $50\%$  minority group.
(a) Minority Group Size:  $25\%$
(b) Minority Group Size:  $50\%$
Figure S6: Performance of CLASH in a simulation experiment with Gaussian outcomes, five covariates, and a minority group that forms (a)  $25\%$  or (b)  $50\%$  of all participants. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint.
H.1.4 Comparison with SUBTLE
Fig. 3 in the main text compares FLASH's performance to that of SUBTLE in when the treatment benefits the majority group and harms (or has no effect on) the minority. We present an analogous comparison in the case when the treatment has no effect on the majority group in Fig. S7. In most cases, FLASH increases the stopping probability over the SUBTLE baseline, especially at the first two interim checkpoints. The only exception is the setting with 10 covariates and a  $12.5\%$  minority group, when FLASH and SUBTLE perform similarly.
(b) Minority Group Size:  $25\%$
Figure S7: FLASH's performance improvement over the SUBTLE baseline with Gaussian outcomes, when the treatment has no effect on the majority group. We plot the difference in stopping probability between FLASH (with an mSPRT) and SUBTLE with  $95\%$  CIs, where the minority group forms (a)  $12.5\%$  or (b)  $25\%$  of all participants.
H.1.5 Comparison with Homogeneous baseline
We now compare FLASH to the homogeneous baseline across a wide range of simulation settings. Fig. S8 compares the two methods when the treatment benefits the majority, while Fig. S9 compares them when the treatment has no effect on the majority. FLASH increases stopping probability in almost all cases.
(b) Minority Group Size:  $25\%$
Figure S8: FLASH's performance improvement over the homogeneous baseline with Gaussian outcomes, when the treatment benefits the majority group. We plot the difference in stopping probability between FLASH (with an OF-adjusted z-test) and the homogeneous baseline with  $95\%$  CIs, where the minority group forms (a)  $12.5\%$  or (b)  $25\%$  of all participants.
(a) Minority Group Size:  $12.5\%$
Figure S9: FLASH's performance improvement over the homogeneous baseline with Gaussian outcomes, when the treatment has no effect on the majority group. We plot the difference in stopping probability between FLASH (with an OF-adjusted z-test) and the homogeneous baseline with  $95\%$  CIs, where the minority group forms (a)  $12.5\%$  or (b)  $25\%$  of all participants.
H.1.6 Small sample size
We evaluate FLASH's performance with much smaller sample sizes ( $N = 200, 400, 1000$ ). We consider an experiment in which the treatment harms the minority group and weakly benefits the majority (i.e.,  $\theta_0 = -0.1$ ). There are five covariates and the minority group forms  $12.5\%$  of the population. Fig. S10 presents results for FLASH with an OF-adjusted z-test. FLASH outperforms the homogeneous baseline by a wide margin in experiments with moderately small samples ( $N = 1000$ ). While the gap between FLASH and the homogenous baseline decreases as sample size decreases, FLASH still outperforms the baseline with as few as 400 participants. However, with very small samples ( $N = 200$ ), neither FLASH nor the homogeneous baseline is likely to stop the experiment.
Figure S10: FLASH's performance in simulation experiments with small samples sizes (N=200, 400, 1000). We consider experiments with Gaussian outcomes, five covariates, a weakly benefitted majority group, and a  $12.5\%$  harmed minority group. We simulate 1,000 trials and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of FLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.7 Large number of covariates