SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recall that an effective method of heterogeneous early stopping must 1) have high stopping probability if any group of participants is harmed and 2) limit the rate of unnecessary stopping. FLASH achieves both of these criteria by constructing weights  $\hat{w}_i^n$  that converge in probability to the optimal weights  $w_i^* \forall i$  (Thm. 3.2). Thus, for  $n$  large enough, FLASH is comparable to a stopping test run only on the harmed population  $S_n^*$  and hence has high stopping probability in the presence of harm (Prop. 3.1). Further, if no group is harmed, the FLASH weights converge quickly to zero. Hence, the test statistic used in Stage 2 converges to zero, as does the probability of unnecessary stopping (Thm. 3.3). Thus, FLASH has high statistical power and limits the Type I error rate relative to the null hypothesis (2).
Our first theorem (proved in App. E.2) establishes the convergence of the CLASH weights to the optimal weights. It makes mild assumptions on the quality of CATE estimation in Stage 1. The choice of CATE estimator may demand further assumptions on the data; for example, using a linear regression provides suitable estimates if  $\tau(x)$  is linear in  $x$ . Alternative causal machine learning methods minimize these assumptions; for example, a causal forest provides sufficiently accurate estimates under weak regularity conditions (e.g., bounded  $x$  [38, Thm. 4.1]).
Theorem 3.2 (CLASH weights converge to optimal weights). CLASH weights (4) satisfy the error bound
$$
| \hat {w} _ {i} ^ {n} - w _ {i} ^ {*} | \leq \exp \left\{- \frac {(\tau (x _ {i}) - \delta) ^ {2}}{2 \hat {\sigma} _ {n , - i} ^ {2} (x _ {i})} \right\} + \frac {| \hat {\tau} _ {n , - i} (x _ {i}) - \tau (x _ {i}) |}{\hat {\sigma} _ {n , - i} (x _ {i})} \exp \left\{- \frac {(\tau (x _ {i}) - \delta) ^ {2}}{2 \hat {\sigma} _ {n , - i} ^ {2} (x _ {i})} + \frac {| \tau (x _ {i}) - \delta | | \hat {\tau} _ {n , - i} (x _ {i}) - \tau (x _ {i}) |}{\hat {\sigma} _ {n , - i} ^ {2} (x _ {i})} \right\}.
$$
Moreover, if  $\delta < \inf_{x:\tau(x) > 0} \tau(x)$  and, given  $x_i$ ,  $\hat{\tau}_{n,-i}(x_i) \stackrel{p}{\to} \tau(x_i)$  and  $\hat{\sigma}_{n,-i}(x_i) \stackrel{p}{\to} 0$ , then  $\hat{w}_i^n$  is a consistent estimator of the optimal weight:  $\hat{w}_i^n - w_i^* \stackrel{p}{\to} 0$ .
Thm. 3.2 provides important intuition on when CLASH will perform well. For observation  $i$ , the rate of weight convergence depends on two factors. First, it is proportional to  $\tau(x_i)$ , the size of the harmful effect. The more harmful the treatment, the better CLASH will perform. Second, it is inversely proportional to  $\hat{\sigma}_{n,-i}(x_i)$ , the uncertainty in the estimate of  $\tau(x_i)$ . The more certain the estimate, the better CLASH will perform. Many factors impact  $\hat{\sigma}_{n,-i}(x_i)$ , including  $n$  and the frequency of observing covariates like  $x_i$ . We expect CLASH to perform better with (1) larger datasets, (2) fewer covariates, and (3) larger  $S_n^*$ . We explore these relationships further in our experiments (Sec. 4-5).
We emphasize that Thm. 3.2 implies a fast rate of convergence, as the error bound decays exponentially in  $1 / \hat{\sigma}_{n, - i}^{2}(x_{i})$ . For example, if  $\hat{\sigma}_{n, - i}^{2}(x_{i}) = o_{p}(1 / \log n)$ , then  $|\hat{w}_i^n -w_i^* | = o_p(1 / n)$  (see Cor. F.1). This is extremely fast: weights and other nuisance parameters are often estimated at a rate of  $1 / \sqrt{n}$  or slower [6]. This speed justifies our use of the Gaussian CDF in (4). While other CDFs can provide Thm. 3.2's consistency result, they may not demonstrate such provably fast convergence. Note that the additional assumption on  $\hat{\sigma}_{n, - i}^{2}(x_{i})$  is mild, as it just needs to decay faster than  $1 / \log n$ . For example, a linear regression under standard assumptions (as in [36]) has  $\hat{\sigma}_{n, - i}(x_i) = \mathcal{O}_p(1 / \sqrt{n})$ .
Our next theorem (proved in App. E.3) establishes that FLASH limits the rate of unnecessary stopping for the OF-adjusted z-test. If no group of participants is harmed, FLASH's stopping probability converges to zero. The rate of unnecessary stopping (i.e., Type I error rate relative to (2)) is thus guaranteed to be  $\leq \alpha$  for  $n$  large enough. The theorem requires a mild condition on the decay of  $\max_{i\leq n}\hat{\sigma}_{n, - i}^2 (x_i)$  (the same discussed above). It also assumes bounded outcomes, a condition satisfied in most real experiments. Thm. F.2 provides a similar result for an SPRT stopping test.
Theorem 3.3 (CLASH limits unnecessary stopping). Consider a stopping test with weighted  $z$ -statistic (3) and weights estimated using CLASH. If  $\max_{i\leq n}\hat{\sigma}_{n, - i}^2 (x_i) = o_p(1 / \log (n))$ ,  $\max_{i\leq n}|\tau (x_i) - \hat{\tau}_{n, - i}(x_i)| = o_p(1)$ , and  $y_{i}$  are uniformly bounded, then the stopping probability of the test converges to zero if no participant group is harmed.
4See [6] for a discussion of the rate of convergence of machine learning-based CATE estimators.
We note that our presented theory is asymptotic, not exact. However, many stopping tests—both frequentist and sequential—rely on asymptotic approximations and so large enough sample sizes are often already required. For example, the mSPRT stopping test with binary outcomes in [14] requires a large-sample Gaussian approximation. In our simulation experiments (Sec. 4) and real-world application (Sec. 5), we show that sample sizes typical in many clinical trials and A/B tests are sufficient for FLASH to improve stopping probability while limiting unnecessary stopping.
3.4 The Decision to Stop
Before proceeding to our experimental evaluation, we emphasize that CLASH is not intended to automate human decisions on early stopping. When CLASH indicates that an experiment should be stopped, the investigator can make one of several decisions. If the nature of harm is serious (e.g., mortality in a clinical trial), the investigator may decide to stop the experiment for all participants. For milder harms (e.g., high latency during application startup in an A/B test), they may decide to stop the experiment only for the harmed group and continue it for all other participants. If the identified harm is much less consequential than the potential benefit (e.g., harm of increased headaches vs. benefit of curing cancer), the investigator may decide to not stop the experiment at all. The specific decision must depend heavily on the ethical and financial aspects of the experiment, a thorough review of the interim data, guidance from the investigator's institutional review board, and other relevant factors.
If the investigator chooses to stop the experiment only on the harmed group, they face two practical challenges: identifying the harmed group and ATE estimation. To identify the harmed group, investigators can either use existing subgroup identification methods (e.g., [1, 30, 44]) on the observed outcomes or analyze the FLASH weights to find covariate combinations with weights close to 1. To estimate the whole population ATE at the end of the experiment, investigators can use inverse propensity weights; this approach allows one to correct for the selection bias induced by stopping the experiment only in one group (see example in Tab. S6). Further details, including ensuring actionable group sizes, are discussed in App. G.
4 Simulation Experiments
We now assess the performance of FLASH in an extensive series of simulation experiments. We focus on two important types of experimental outcomes: Gaussian and time-to-event (TTE). Gaussian outcomes are the most frequently considered (e.g., [14]), as they are common in both clinical trials and A/B tests. Many stopping tests assume either that the data is Gaussian or that the average outcome is asymptotically Gaussian. Meanwhile, TTE outcomes are very common in clinical trials. We establish that FLASH improves stopping times over relevant baselines in both these scenarios. Our real-world application (Sec. 5) then applies FLASH to outcomes that are approximately negative binomial, further demonstrating that FLASH can be impactful in experiments across several domains.
Setup: Gaussian Outcomes We consider a randomized experiment that evaluates the effect of  $D$  on  $Y$ . Participants come from two groups, with  $G \in \{0,1\}$  indicating membership in a minority group. We do not observe  $G$ , but observe a set of binary covariates  $X = [X_1,..,X_d]$ , where  $d$  varies between 3 and 10 and  $p(X_j = 1) = 0.5 \forall j$ .  $X$  maps deterministically to  $G$ , with  $G = \prod_{j=1}^{k} X_j$ . We vary  $k$  between 1 and 3: the expected size of the minority thus varies between  $12.5\%$  and  $50\%$  (of all participants).  $Y$  is normally distributed within each group, with  $Y|G = 0 \sim \mathcal{N}(\theta_0D,1)$  and  $Y|G = 1 \sim \mathcal{N}(\theta_1D,1)$ . The distribution of  $Y$  over the whole population is thus a mixture of two Gaussians. We vary  $\theta_0$  between 0 and -0.1; the majority group is thus unaffected or benefited by the treatment. We vary  $\theta_1$  between 0 and 1: the minority group is thus either unaffected or harmed.
The experiment runs for  $N = 4000$  participants and  $T = 2000$  time steps, recruiting one treated and one untreated participant at each step. The experiment has three checkpoints, with 1,000, 2,000, and 3,000 participants (corresponding to  $25\%$ ,  $50\%$ , and  $75\%$  of the total time duration). At each
5Replication code for the simulation experiments can be found at https://github.com/hammaadam1/ clash
This is a realistic setting (e.g.,  $X$  is device type and new feature  $D$  increases application latency  $Y$  only on some devices). However, FLASH also performs well if the mapping from  $\mathbf{X}$  to  $\mathbf{G}$  is stochastic (see App. H.1.9).
This assumption is required for some stopping tests (e.g., mSPRT, see [14]), not for FLASH. FLASH can be used with any experiment design (e.g. batch recruitment, imbalanced arms) if a suitable stopping test is used.
Figure 2: Performance of FLASH in a simulation experiment with Gaussian outcomes. If the minority group is harmed, FLASH (red) significantly increases the stopping probability over the homogeneous baseline (blue) and SUBTLE (pink). For large effect sizes, FLASH is as effective as an oracle that has prior knowledge of the harmed group (green). Crucially, if the treatment has no harmful effect (i.e., effect = 0), FLASH does not stop the trial more often than either baseline (i.e., FLASH limits Type I error rate). These results hold for four commonly-used stopping tests (corresponding to each column). SUBTLE builds upon the mSPRT framework specifically, so is only presented in the second column. These results hold whether the majority group's treatment is beneficial (top row) or has no effect (bottom row). We simulate 1,000 trials with 4,000 participants each and a  $12.5\%$  minority group; we plot the stopping probability (with  $95\%$  CIs) at any interim checkpoint.
checkpoint, we run CLASH and compute its stopping probability across 1,000 replications. In Stage 1, CLASH uses a causal forest with 5-fold CV and  $\delta = 0.1$ . Standard errors were estimated by the causal forest itself (i.e., not via the bootstrap). In Stage 2, CLASH uses one of four commonly-used stopping tests: an OF-adjusted z-test, an mSPRT [14], a MaxSPRT [16], and a Bayesian estimation-based test.
We compare FLASH's stopping probability to three alternative approaches (two baselines and one oracle). The homogeneous baseline applies the four aforementioned stopping tests to the collected data in aggregate. The SUBTLE baseline uses a recently-developed heterogeneous stopping test [43]. SUBTLE is the only existing approach that can handle unknown groups without strong parametric assumptions (e.g., linearity). SUBTLE builds on the mSPRT framework and has the same decision rule; we thus compare its performance to FLASH using mSPRT in Stage 2. The Oracle applies the four aforementioned stopping tests only to data from the harmed group. This approach reflects the optimal test discussed in Prop. 3.1, and represents an upper bound on performance.
Setup: TTE Outcomes We consider a clinical trial that measures time to a positive event (e.g., remission) and defines treatment effects using the hazard ratio. We adapt the simulation setup from [20]; see App. H.2.1 for a full description. FLASH's Stage 1 uses a survival causal forest [8] and Stage 2 uses an OF-adjusted Cox regression. Note that existing SPRTs cannot be applied to TTE outcomes, as they cannot account for repeated observations for each participant. Crucially, SUBTLE cannot be used either; FLASH is thus the first heterogeneous method applicable to this setting.
Results CLASH outperforms both the homogeneous baseline and SUBTLE in a wide range of experiments. We first discuss performance in the Gaussian setting with five covariates and a minority group that represents  $12.5\%$  of participants. Fig. 2 plots the probability that the experiment is stopped at any of the three checkpoints. If the minority group is harmed (x-axis  $\theta_{1} > 0$ ), CLASH stops the experiment significantly more often than the homogeneous and SUBTLE baselines. The magnitude of the improvement depends on the effect size: the larger the effect, the better CLASH performs. For large effects, CLASH even converges to the oracle. Crucially, CLASH does not increase the rate of unnecessary stopping: if the minority group is unharmed ( $\theta_{1} = 0$ ), CLASH stops the experiment no more frequently than either baseline (i.e., CLASH controls the Type I error rate). These results
hold for all four stopping tests; CLASH thus leads to effective stopping no matter which test is used. Moreover, its performance gains are robust to the specific choice of the hyperparameter  $\delta$  (Fig. S5).
We now compare FLASH to SUBTLE, the heterogeneous baseline presented in the second column of Fig. 2. FLASH improves stopping probability over SUBTLE whether the treatment benefits the majority group or has no effect. We focus on the former case and plot the increase in stopping probability of FLASH (using mSPRT) over SUBTLE in Fig. 3. FLASH improves stopping probability in a large number of scenarios. We make three specific notes. First, when the minority group is larger, FLASH performs well for a broader range of effect sizes. Notably, SUBTLE greatly underperforms FLASH for medium effects (though it is able to detect large effects as often as FLASH). Second, increasing the number of covariates only has a small impact on FLASH, which outperforms SUBTLE despite the higher dimensionality. Third, FLASH greatly increases the probability of stopping the trial at the first and second checkpoints. FLASH thus not only stops trials more often, but also faster.
CLASH also demonstrates strong performance with TTE outcomes (App. H.2). CLASH improves the stopping probability of the OF-adjusted Cox regression by 20 percentage points if the treatment harms  $25\%$  of the population with hazard ratio of 0.7. No existing method for heterogeneous stopping (including SUBTLE) can handle TTE; this broad applicability is one of CLASH's major advantages.
App. H presents detailed simulation results that further demonstrate FLASH's efficacy. We note the discussed results are robust across a range of simulation settings, including effect size, sample size, covariate dimensionality, harmed group size, and outcome variance. Furthermore, FLASH is highly effective in situations with multiple harmed groups, noisy group membership, and unknown outcome variance. While FLASH broadly outperforms existing methods, we note two specific limitations. First, FLASH may struggle to detect harmful effects in very small samples (e.g.,  $N = 200$ , see Fig. S10). Note that this is a challenging setting for any method, since we only observe 10-20 data points from the harmed group at any interim checkpoint. Second, FLASH may struggle with a very large number of covariates (e.g., 500 covariates, see Fig. S11). In such settings, we recommend performing feature selection before running FLASH.
5 Real-world Application
In this section, we apply CLASH to real data from a digital experiment. We consider an A/B test run by a technology company to evaluate the effect of a software update on user experience. The dataset was collected in 2022 and consists of 500,000 participants, roughly half of whom received the update.
(a) Minority Group Size:  $12.5\%$
(b) Minority Group Size:  $25\%$
Figure 3: FLASH's performance improvement over the SUBTLE baseline with Gaussian outcomes. FLASH significantly increases stopping probability over SUBTLE across a range of effect sizes (x-axis) and covariate dimensions (rows). FLASH not only stops more often but also faster, with higher stopping probability at earlier checkpoints (columns). FLASH does not increase stopping probability if the treatment is not harmful (i.e., effect = 0). We plot the difference in stopping probability between FLASH and SUBTLE (with 95% CIs), where the minority group forms (a) 12.5% or (b) 25% of all participants. For the larger group size, FLASH's improvement is most notable for medium effects, which SUBTLE struggles to detect. All performance increases are robust to an increase in the number of covariates. See App. H.1 for a similar comparison of FLASH to the homogeneous baseline.
We evaluate this treatment's effect on a proprietary metric that measures negative user experience. An increase in the outcome—which is discrete and right-skewed—indicates a worse user experience. The experiment also recorded which one of eight geographic regions the user's device was located in. We first use the full dataset to assess whether the treatment had a heterogeneous impact by region. Separate negative binomial (NB) regressions indicate that the treatment led to a large statistically significant increase in the metric in Region 1, a small significant increase in Regions 2, 3, and 4, no significant effect in Regions 5 and 6, and a significant decrease in Regions 7 and 8 (Tab. S3).
We evaluate stopping tests on this dataset, conducting checkpoints every 20,000 participants. We run CLASH with a causal forest and  $\delta = 1$  in Stage 1 and an OF-adjusted NB regression in Stage 2. We find that CLASH decreases stopping time over the homogeneous baseline, stopping the experiment after 40,000 as opposed to 60,000 participants. SUBTLE stops after 80,000 participants; this under-performance may result from the data's strong non-normality. An oracle that only considers Region 1 also stops after 40,000 observations (i.e., same as CLASH). We note that CLASH's optimal performance is not a fluke: across 1,000 random shuffles of the dataset, CLASH stops the experiment at the same interim checkpoint as the Oracle in  $62.6\%$  of shuffles (details in Tab. S5).
We emphasize that by stopping the experiment early, the company can minimize customer frustration, preventing churn and financial impact. Domain expertise can guide whether to stop the experiment altogether, or just in the harmed regions. The harmed regions can be identified at stopping time using separate regressions (Tab. S4), FLASH weights (Fig. S19), or policy learning [15, 1]. Note that stopping the experiment just in one region would affect statistical inference at the end of the experiment, as the treatment would no longer be randomly assigned across regions. As described in Sec. 3.4, investigators can use inverse probability weighting to adjust for this selection (Tab. S6).
Figure 4: FLASH performance by sample size on real data. We plot stopping probability across 1,000 experiments with  $95\%$  CIs.
Finally, we study the impact of sample size on FLASH's performance. We vary sample size  $N_{s}$  between 10,000 and 100,000, and generate 1,000 experiments for each  $N_{s}$  by sampling from our dataset. We only sample from Region 1 and Regions 5-8; this gives us one harmed group (Region 1) that comprises  $29\%$  of the total population (see App. I for results with Regions 2-4). We conduct one checkpoint at the halfway stage of each simulated experiment. We find that FLASH significantly improves stopping probability over the homogeneous baseline for all  $N_{s} \geq 40,000$ , and converges to near-oracle performance around  $N_{s} = 90,000$  (Fig. 4). These sizes represent  $8\%$  and  $18\%$  of the total sample of the A/B test. This result is encouraging, as it indicates that datasets that are a fraction of the
size of typical online A/B tests are large enough for our asymptotic theory (Sec. 3.3) to hold.
We propose a new method CLASH for the early stopping of randomized experiments on heterogeneous populations. CLASH stops experiments more often than existing approaches if the treatment harms only a subset of participants, and does not stop experiments if the treatment is not harmful. Prior work is either limited by restrictive assumptions (e.g., linearity) or incompatible with common data distributions (e.g., time-to-event). In contrast, CLASH is easy to use, adapts any existing stopping test, and is broadly applicable to clinical trials and A/B tests. Our work has a few important limitations. First, CLASH works better with a relatively small number of covariates. While this is not a problem for most experiments—clinical trials and A/B tests typically collect only a few covariates—a different pre-processing approach may be required in high-dimensional settings (e.g., genetic data with over 500 covariates). Second, CLASH may not detect harm in experiments with a very small number of participants. For example, sample sizes typical in Phase 1 clinical trials (i.e., less than 100 participants) may be too small for CLASH to be more effective than existing approaches. Finally, this paper only considers stopping for harm. In practice, experiments are often stopped for early signs of benefit and futility. Expanding CLASH to these decisions is an important direction for future work.
We thank Steven Goodman, Sanjay Basu, Dean Eckles, Nikos Trichakis, and Sonia Jaffe for their helpful comments. This work was graciously supported through the Bowers CIS Strategic Partnership Program with LinkedIn.