SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate FLASH's performance with high-dimensional covariate sets (100 and 500 covariates). We consider an experiment with 4,000 participants in which the treatment harms a  $12.5\%$  minority group and weakly benefits the majority (i.e.,  $\theta_0 = -0.1$ ). Fig. S11 presents results for FLASH with an OF-adjusted z-test. FLASH is robust to increasing dimensionality to a point, outperforming the homogeneous baseline even with 100 covariates. The extreme case with 500 covariates is more challenging: here, investigators may need to perform feature selection before running FLASH.
Figure S11: CLASH's performance in simulation experiments with high-dimensional covariate sets (100 and 500 covariates). We consider experiments with Gaussian outcomes, a weakly benefitted majority group, and a  $12.5\%$  harmed minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.8 Multiple harmed groups
We evaluate FLASH's performance with multiple  $(>2)$  groups in the population. We refer to the x-axis effect size values as  $x$ , and consider two settings:
a) Three groups of unequal size (group size and effect size, respectively, in parentheses): one weakly benefited  $(87.5\%, -0.1)$ , one strongly harmed  $(6.25\%, x)$ , and one weakly harmed  $(6.25\%, x/2)$ .
b) Four equally sized groups (effect sizes in parentheses): strongly benefited  $(-x)$ , weakly benefited  $(-x/2)$ , weakly harmed  $(x/2)$ , and strongly harmed  $(x)$ .
Fig. S12 displays the results. Overall, FLASH performs well: it stops more frequently than the homogeneous baseline across a range of effect sizes and as often as the Oracle for larger effects.
(a) 3 groups
(b) 4 groups
Figure S12: FLASH's performance in simulation experiments with multiple harmed groups. We consider experiments with Gaussian outcomes, five covariates, and (a) three groups (one weakly benefited majority group, one weakly harmed  $6.25\%$  minority group, one strongly harmed  $6.25\%$  minority group), or (b) four equally sized groups (one strongly benefited, one weakly benefited, one weakly harmed, one strongly harmed). We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of FLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.9 Stochastic group membership
In previous experiments, the mapping from covariates to groups was deterministic. We now evaluate CLASH's performance in situations where the covariates map stochastically to the benefited and harmed groups. We construct a  $25\%$  minority group deterministically from covariates as before, but randomly assign  $p\%$  of this group to be harmed. The remainder of the minority group and the majority group are both weakly benefited with effect size  $\theta_0 = -0.1$ . We consider experiments with  $N = 4,000$  and five covariates, and plot results in Fig. S13. CLASH outperforms the homogeneous baseline both when  $p = 0.5$  and  $p = 0.75$ .
Figure S13: CLASH's performance in simulation experiments with stochastic group membership. We consider experiments with Gaussian outcomes, five covariates, and a weakly benefitted majority group.  $p\%$  of participants from a  $25\%$  minority group are harmed by the treatment; the rest are weakly benefitted. We vary  $p$  between 0.5 and 0.75. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.10 Smaller / larger outcome variance
We evaluate CLASH's performance as the variance in the outcomes increases. We consider an experiment with 4,000 participants and five covariates in which the treatment harms a  $12.5\%$  minority group and weakly benefits the majority  $(\theta_0 = -0.1)$ . We vary  $\operatorname{Var}(Y)$  between 0.5, 1, and 2 (recall that  $\operatorname{Var}(Y) = 1$  in previous experiments). Fig. S14 demonstrates that CLASH outperforms the homogeneous baseline in all settings. CLASH and the Oracle both perform better with lower variance.
Figure S14: FLASH's performance in simulation experiments with smaller and larger variance in the observed outcomes. We consider experiments with Gaussian outcomes, five covariates, a weakly benefitted majority group, and a harmed  $12.5\%$  minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of FLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.11 Unknown outcome variance
We now consider a setting in which the variance in the outcomes is unknown. The setup is exactly the same as in App. H.1.10, except that now we do not know  $\operatorname{Var}(Y)$ . Instead, we estimate the variance from the observed outcomes and use the plug-in version of the z-test. Fig. S15 demonstrates that while estimating the variance has a small effect on CLASH's performance, CLASH still drastically outperforms the homogeneous baseline.
Figure S15: FLASH's performance in simulation experiments with unknown outcome variance. We consider experiments with Gaussian outcomes, five covariates, a weakly benefitted majority group, and a harmed  $12.5\%$  minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint of FLASH (with an OF-adjusted z-test with plug-in variance), the homogeneous baseline, and the Oracle.
H.2 TTE Outcomes
H.2.1 Setup
We consider a clinical trial that measures time to a positive event (e.g., remission), and adapt the simulation setup from [20]. The trial runs for 30 months and recruits 2,000 participants, who accrue uniformly at random over the study's first year. Some participants drop out of the trial: drop out time follows an exponential distribution with an annual hazard rate of 0.014. Group membership and covariates are generated as in the Gaussian case. The treatment effect is measured using the hazard ratio (HR), where  $HR > 1$  is beneficial and  $HR < 1$  is harmful. Outcomes follow a survival function with  $S(t) = \exp \{-0.1(1 - D)t - \theta_0D(1 - G)t - \theta_1DGt\}$ , with  $\theta_{1} \in [0.05, 0.1]$  and  $\theta_0 \in \{0.1, 0.12\}$ . The treatment thus either benefits or has non-effect on the majority group, and either harms or has no effect on the minority group.
We conduct checkpoints at 12, 18, and 24 months.  ${}^{8}$  FLASH's Stage 1 uses a survival causal forest [8] and  $\delta  = {0.05}$  . Stage 2 uses an OF-adjusted Cox proportional hazards regression. Note that SPRT-based tests have not yet been adapted to the TTE setting; among other reasons, the statistical dependence between checkpoints (induced by observing additional data from the same participant) proves challenging for these tests. Crucially, SUBTLE cannot be used either; FLASH is thus the first heterogeneous stopping method applicable to this setting. We compute FLASH's stopping probability at each checkpoint across 1,000 replications, and compare it to the homogeneous baseline and oracle.
As with the Gaussian experiments, each replication uses its own random seed. One replication—including all combinations of effect size, minority group size, number of covariates and stopping tests—takes between 137 and 202 minutes to run on a single CPU (depending on the random seed). This yields a total compute time of approximately 3,000 hours for the TTE experiments. The simulations were run in parallel on an academic computing cluster with over 200 CPUs. Note that each replication used one CPU with 4GB of RAM.
Figure S16: Performance of FLASH in a simulation experiment with TTE outcomes. If the minority group is harmed (lower hazard ratio), FLASH (red) significantly increases the stopping probability over the homogeneous approach (blue). This improvement is most notable when then treatment benefits the majority group. We simulate 1,000 trials with 2,000 participants and five covariates and plot the stopping probability (with  $95\%$  CI) at any interim checkpoint.
<sup>8</sup>We define checkpoints in terms of time, as TTE trials update observations for the same participants.
H.2.2 Results
CLASH (with an OF-adjusted Cox regression) significantly improves stopping probability over the homogeneous baseline if the treatment benefits the majority group and harms the minority (Fig. S16). For a  $25\%$  minority group size, this performance improvement is most notable for large effect sizes  $(HR \leq 0.6)$ , while for a  $50\%$  minority group size, the improvement is most notable for medium effects  $(HR$  between 0.7 and 0.8). CLASH not only stops more often, but also faster, with higher stopping probability at earlier interim checkpoints (Fig. S17). These performance improvements are reduced when the treatment has no effect on the majority group (Fig. S18). Note that we do not consider harmed group sizes smaller than  $25\%$ , as no considered method was able to detect harmful effects with TTE outcomes on such small populations.
(b) Minority Group Size:  $50\%$
Figure S17: FLASH's performance improvement with TTE outcomes when the treatment benefits the majority group. FLASH significantly increases the stopping probability across a range of harmful effect sizes. FLASH not only stops more often, but also faster, boosting the stopping probability at earlier checkpoints. We plot the difference in stopping probability between FLASH and the homogeneous baseline (with  $95\%$  CIs), where the minority group forms (a)  $25\%$  or (b)  $50\%$  of all participants. The treatment effect is given by the hazard ratio (smaller HR indicates greater harm). For the larger group size, FLASH's improvement is most notable for medium effects, as the homogeneous baseline is also able to detect large effects. All performance boosts are robust to an increase in the number of covariates.
(a) Minority Group Size:  $25\%$
Figure S18: FLASH's performance improvement with TTE outcomes when the treatment has no effect on the majority group. We plot the difference in stopping probability between FLASH and the homogeneous baseline (with  $95\%$  CIs), where the minority group forms (a)  $25\%$  or (b)  $50\%$  of all participants. The treatment effect is given by the hazard ratio (smaller HR indicates greater harm).
I Real-world Application
We now present supplementary tables and figures that were referenced in our discussion of the real-world application in Sec. 5. We first present regional treatment effects, estimated with the whole dataset (Tab. S3) and at FLASH's stopping time (Tab. S4). We then present results from our semi-synthetic evaluation of FLASH's performance by sample size. In Fig. 4 in the main text, we only sampled from Region 1 (harmed group) and Regions 5-8 (unharmed group); this subset yielded a harmed group that comprised  $29\%$  of the total population. We now increase the size of the harmed group by including data from Regions 2-4. Note that we considered a region as harmed if the treatment effect estimated with the whole dataset (Tab. S3) was positive and significant; all other regions were considered unharmed. Further note that Fig. 4 in the main text only included Region 1 (instead of one of Regions 2, 3, or 4) since it was the most harmed region (Tab. S3). The results of this semi-synthetic evaluation (Fig. S20) are similar to those discussed in Sec. 5.
Table S3: Regional Effect of Treatment on Performance Metric (estimated at experiment's end with 500,000 observations). We present coefficients from separate univariate negative binomial regressions.
Treatment Effect Estimate	Std. Error	p-value
Region 1	0.385	0.014	0.000
Region 2	0.065	0.016	0.000
Region 3	0.107	0.034	0.002
Region 4	0.056	0.010	0.000
Region 5	0.006	0.015	0.684
Region 6	0.014	0.017	0.424
Region 7	-0.048	0.021	0.023
Region 8	-0.311	0.025	0.000
Table S4: Regional Effect of Treatment on Performance Metric (estimated at checkpoint with 40,000 observations, i.e., when FLASH stops the experiment). We present coefficients from separate univariate negative binomial regressions.
Treatment Effect Estimate	Std. Error	p-value
Region 1	0.453	0.051	0.000
Region 2	0.376	0.051	0.000
Region 3	0.250	0.130	0.054
Region 4	-0.146	0.036	0.000
Region 5	-0.115	0.065	0.078
Region 6	0.260	0.055	0.000
Region 7	-0.253	0.077	0.001
Region 8	-1.292	0.083	0.000
Table S5: Mean stopping time in our empirical application. We shuffle our real-world dataset 1,000 times and compute the mean stopping time (with standard errors) for FLASH, the homogeneous baseline, and the Oracle.
Method	Mean Stopping Time (Std. Error)
Oracle	48,500 (431)
CLASH	57,200 (609)
Homogeneous	64,420 (848)
Table S6: ATE estimation in our empirical application if the experiment is only stopped for the harmed group. We assume that the experiment is stopped in Region 1 at FLASH stopping time of 40,000 participants and continued in all other regions. We consider the ATE without stopping (i.e., after continuing to collect data from all regions) as the ground truth. The naive ATE estimate (i.e., an unweighted estimate using all collected data) is biased and underestimates the true ATE. Using the IPW approach detailed in App. G corrects this bias.
Estimate (Std. Error)
ATE estimate without stopping	0.11 (0.006)
Naive ATE estimate with stopping	0.03 (0.006)
IPW ATE estimate with stopping	0.10 (0.006)
Figure S19: Distribution of FLASH-estimated weights by region (at checkpoint with 40,000 observations, i.e., when FLASH stops the experiment). FLASH identifies that the treatment has a strong harmful effect in Regions 1 and 2 and assigns these observations unit weight.
(a) Region 1
(b) Regions 1-2
(c) Regions 1-3
(d) Regions 1-4
Figure S20: FLASH's performance by sample size with real data from an A/B test. We plot stopping probability across 1,000 random samples with  $95\%$  CIs. We define the unharmed group as Regions 5-8. We define the harmed group as (a) Region 1 ( $29\%$  harmed group size), (b) Regions 1-2 ( $41\%$  harmed group size), (c) Regions 1-3 ( $43\%$  harmed group size), or (d) Regions 1-4 ( $60\%$  harmed group size). In cases (a) - (c), FLASH significantly increases stopping probability over the homogeneous baseline for all  $N_{s} \geq 50,000$ , and achieves near-oracle level performance with  $N_{s} = 100,000$  ( $20\%$  of the overall experiment's size). Case (d) reflects a situation in which a majority group is harmed, which is not our method's primary focus. However, FLASH still outperforms the homogeneous baseline by  $N_{s} = 100,000$ .